Integrate Metascraper crawl to operate on the Puppeteer crawler output

[kstapelfeldt]

• A single JSON is created for every successful link crawled in puppeteer
• Metascraper portion needs to read these JSONs, grab the domain name, and then try either appending the data to the existing JSON or create a separate directory with the new JSONs (to be merged later on).
• How do we keep track of what's already been looped? Suggestion: Rename file using some kind of reserved character (something that would not appear normally with the UUID) so that we can see what's been processed (for example, append a "-M) OR see whether or not we could use a database and add a column that will allow us to track progress. -If we use a DB, then the post-processor would have to read from the DB. We could read Raiyan's files, or modify puppeteer to write to database. Need to research best solution. Review https://nodejs.org/api/fs.html for possible tools

[kstapelfeldt]

• Work has started and it moves steadily.
• However, Alex will need to work with Jacqueline's new database (bring in her latest branch)
• track if you successfully got the date or not
• contact @jacqueline-chan with any questions re: database.

[kstapelfeldt]

Exam schedule got in the way. No progress this week. Tasks from last week remain.

[kstapelfeldt]

• functioning with database. Reads date from database, makes metascraper crawl, writes date back.
• todo: Grab all metadata and append to the same JSON file with a prefix metascraper_
• Possible future test: Alejandro could put together specific article links to test in metascraper as a mechanism for determining the ongoing relevance of the current scope and Alex will test what metascraper brings back. @jacqueline-chan could dig around for testing script and send to Alejandro/ @AlAndr04 if we decide to do this.
• Rest of ticket is on hold until we have sorted out what is happening with crawler (would affect queue) and post-processor.

[kstapelfeldt]

Extra columns have been added and a pull request needs to be made.

@AlAndr04 has written code for testing metascraper dates.

[kstapelfeldt]

Merged and closing for now.