rmax / dirbot-mysql

Scrapy project based on dirbot to show how to use Twisted's adbapi to store the scraped data in MySQL.
117 stars 55 forks source link

Populating DBs; roxy_ui vs memex_uncharted #4

Open mikew77 opened 8 years ago

mikew77 commented 8 years ago

Not so much of an issue, but more a question about how to best begin getting data into the environment.

I'm looking at getting a scraper / pipeline set up to begin populating the DB. From looking at the schemas, it looks like roxy_ui is for scraped data whereas memex_uncharted is more for rules and context. Is this correct? Any advice would be much appreciated!