Review scraper and CVS import modules

[graphiclunarkid]

These two features need to be reviewed to figure out how much work they would take to bring make functional should our clients request them.

The CVS import module is to automate import of data from CSV files.

The scraper feature is to scrape a web resource to import video data and its metadata.

ACCEPTANCE CRITERIA

• Github issues exist that describe the work required to implement both features.

[ggaughan]

I think the existing monitoring tab:

• loads dummy file counts and error lists from a static file (importer_stats.json)
• loads and tries to save (currently gives 403 csrf failure) config files containing paths for scraping and importing
• some of the paths are validated
• shows options for mysql and job start times but doesn't load or save these anywhere

It also (too) frequently tries to poll (currently fails possibly because of a missing id parameter) a table containing a last-updated-time. I can find no code to do any:

• scraping of data or media
• importing of csv or media
• updating of the last-updated-time or counts/import error statuses

I'd suggest implementing any future import/scraping jobs via Celery tasks and use celery-beat to schedule them if necessary.