Currently, if the web scraper runs into certain errors, we lose all of the data that we just scraped! This obviously isn't desirable, but there are certain errors that would make it impossible to write to the database (such as a database error itself). We probably shouldn't automatically try to commit this data since it could contain errors, so what do we do with it?
Should we dump this data to a file?
To the console?
Would it just be easier to run the scraper again to collect more data?
At least during development, it could be useful to have some sort of data dumping mechanism to help figure out what went wrong.
How should this work?
How would we view this data?
Do we need to create a script that would import this data into the database?
Currently, if the web scraper runs into certain errors, we lose all of the data that we just scraped! This obviously isn't desirable, but there are certain errors that would make it impossible to write to the database (such as a database error itself). We probably shouldn't automatically try to commit this data since it could contain errors, so what do we do with it?
At least during development, it could be useful to have some sort of data dumping mechanism to help figure out what went wrong.