search

ActiveConclusion / COVID19_mobility

COVID-19 Mobility Data Aggregator. Scraper of Google, Apple, Waze and TomTom COVID-19 Mobility Reports🚶🚘🚉
MIT License
268 stars 58 forks source link

Data with dates duplicated? #22

Closed iMarcello closed 4 years ago

iMarcello commented 4 years ago

Hi, first at all thank you very much for share this data! Is a very useful resource. Today I have read the data in a PAndas DataFrame and I have encountered duplications in date however with different data. As an example:
country region date retail and recreation workplaces 199877 Nicaragua Total 2020-02-15 6 4 199878 Nicaragua Total 2020-02-16 4 1 200039 Nicaragua Total 2020-02-15 9 2 200040 Nicaragua Total 2020-02-16 7 -1

Is the first time I see that, could be the original data corrupted? Thanks in advance for your att. Marce

cipolataman commented 4 years ago

Same issue as above - noticed the entry for Egypt had this error.

ActiveConclusion commented 4 years ago

Hi! Thanks for reporting the issue, I will look into it. I suppose it caused by changes in the original Google Mobility reports.

ActiveConclusion commented 4 years ago

So, there are two significant changes in the last update of Google reports: 1) Added metro_area column. It's one more administrative level, for metropolitan areas. 2) Before this update, the second level of subregions (column sub_region_2) was used only for the US counties. Now, there are many countries for which data on this level were added.

It's very good that Google added more data, but from another side, it adds more problems how better and easier to process it. Currently, I don't add data for the second level of subregions and metropolitan areas in this preprocessed Google Mobility report. I decided to do this so as don't ruin report merging. Please, check if everything is ok with the dates now.

I need some time to think about how to deal better with new updates.

iMarcello commented 4 years ago

Hi again, thanks for your help and fast response. No duplications for Nicaragua. I also test Egyot (EG) with good results also. Again, VERY APPRECIATED! Regards. EG_Google_Mobility

cipolataman commented 4 years ago

@ActiveConclusion you legend! Thank you for the quick fix. Super appreciated.