Save scraper task data - Githubissues

0xkash / rest-scraper

A real estate scraper with a built in REST API

1 stars 0 forks source link

Save scraper task data #7

Open 0xkash opened 1 year ago

0xkash commented 1 year ago

When a scraper is set in the task queue I want to track the progress and see when its run. Now only the status in the Scraper object is available, however this does not give enough information about when its run or even if the ran task had any troubles.

0xkash commented 1 year ago

Two options for this:

Maybe just adding a scraped_date_start and scraped_date_completed column to see when the scraper has run.
A new table tasks where the PK is the task uuid. This can also store more then just the dates but also failure response and success response