search

ActiveConclusion / COVID19_mobility

COVID-19 Mobility Data Aggregator. Scraper of Google, Apple, Waze and TomTom COVID-19 Mobility Reports🚶🚘🚉
MIT License
268 stars 58 forks source link

Add Waze data scraper #19

Closed ActiveConclusion closed 3 years ago

ActiveConclusion commented 4 years ago

https://www.waze.com/covid19

ActiveConclusion commented 4 years ago

Waze data was added to the repository. Currently, my script preprocesses raw data automatically, while the raw data will be manually uploaded to the repository by me. Merging with Google and Apple reports will be done later.

ActiveConclusion commented 4 years ago

Automatic downloading of Waze data is a difficult problem because the data can only be obtained from Google Data Studio dashboard. Simple Html-parser does not work here, but this task can be solved with the Selenium library. On GIF below you can see how I can automatically retrieve data using this library. I have to log into my Google account first for the "Download СSV" button to be available on the dashboard. scraper Currently, I use this script for regular update Waze data from my computer. I tried to upload this script to Github to automatically update the data using Github Actions, but predictably it didn't work out. That's why. log aft This step is also possible to pass automatically, but I don't want that simple data scraper looks like some bank robbing software. But I didn't give up and found the second way of data scraping. It's possible to retrieve all data from table view on the dashboard. To do this, it is necessary to automate the pressing of the navigation arrow. right arrow Google login is not required for this solution, so I easily set up this script on Github. I'll keep an eye on how stable this solution is for a while, and if it's okay, I'll use this way of an update for Waze data in this repository.

ActiveConclusion commented 4 years ago

Waze data updating is now automated.

ActiveConclusion commented 3 years ago

Merging with other data is postponed.