search

Sanpele / websitePlaying

Personal website little projects.
0 stars 0 forks source link

Able to scrape all days in bulletin #12

Open Sanpele opened 2 years ago

Sanpele commented 2 years ago

The current approach is to only scrape data from the most recent update. This approach was valid when the data was updated daily however the province of Manitoba has moved to more sporadic updates often doing m/w/f although things understandably were more varied over the holidays.

In regards to the quality of data provided case counts are still provided for other days however test positivity rate is not. Thus partial data could be scraped and a more complete timeline of the daily cases could be constructed.

Dates will need to be inferred from the current date, which is definitely possible but may be a little tricky especially around month boundaries.

Starting point should probably be ignoring month boundaries and implementing a simplified version. Checking for updates for each day with the keyword "on Sat" or "on Mon" ect.

Sanpele commented 2 years ago

Thinking of not using this code to scrape as python approach w/o using a scraping API w/ limited number of calls per month not ideal. Possible pivot for this code -> get / use URL csv of all previous bulletins and scrap / add to DB a random day. Will display functionality and accomplish original purpose and give room to work on different approach for scraping covid numbers.