Open r-lomba opened 4 years ago
noiceeeeeeeeeeeeeeeeeeee
it seems you're missing import numpy as np.
Also line 110: date_time_columnname = yesterday.strftime("%-m/%-d/%Y") was throwing me an error. I took away the dashes and it worked: date_time_columnname = yesterday.strftime("%m/%d/%Y")
I'm now correctly getting worldometer_recovered_2020_03_26.csv correctly in my folder
P.S. I also had to pip install lxml
@ablanch5 sorry, I forgot a few imports. I modified the code above, can you try now?
@ablanch5 the dashes are to avoid padding in the date e.g. NOT "03/25/2020" but instead "3/25/2020" so I used the dashes to be compliant with JHU... Maybe your python version does not support that format?
I'm using python 3.7.6. When I look at the csv file I'm getting 3/25/2020 like you wanted. Could it be a Linux/Windows thing? I don't know
@ablanch5 I don't know. I use Linux. Glad it worked, anyway! Enjoy :)
I think its a windows/linux thing: https://stackoverflow.com/questions/10807164/python-time-formatting-different-in-windows
Nice work!
Something like this would fix the date format issue:
import os
...
...
# MISMATCH IN DATE FORMAT AS DOCUMENTED IN:
# https://stackoverflow.com/questions/10807164/python-time-formatting-different-in-windows
if os.name == 'nt':
date_time_columnname = yesterday.strftime("%m/%d/%Y")
else:
date_time_columnname = yesterday.strftime("%-m/%-d/%Y")
@efra-mx have you tested it? If you have and you sure it works, I'll update the code snippet. Otherwise if not tested, let's just leave your comment here for inspiration for future readers.
In the meanwhile I've updated the code because since today if you don't "pretend to be a browser" they give you a 403 error. Now the problem is fixed (I don't get why they try to protect a public HTML table on a public page, in times of pandemic... Nevertheless, for now it still works with the updated code)
@r-lomba I don't have a windows machine myself. But you can check: https://stackoverflow.com/questions/1325581/how-do-i-check-if-im-running-on-windows-in-python https://stackoverflow.com/questions/1854/what-os-am-i-running-on
Good job by the way.
I wanted to share a refactored version of the code done by @r-lomba. It is wrapped in a class It will be easier to use.
r = CovidReader() r = fetch() print(r.recovered_by_country())
I hope you find it helpful
import os
import pandas as pd
import numpy as np
import dateutil
import time
import copy
from datetime import datetime, timedelta, date
import requests
#############################################################
##
## HARDCODED DICTIONARIES - FIXES TO COUNTRIES THAT ARE NAMED
## DIFFERENTLY IN WORLDOMETER STANDARDS THAN JHU STANDARDS
## HERE WE RENAME COUNTRIES IN RECOVERED DATAFRAME FROM WORLDOMETER
## (ON THE LEFT HAND SIDE) TO MAKE THE COUNTRY NAMES MATCH WITH
## THOSE USED AT JHU (ON THE RIGHT HAND SIDE)
##
## PLEASE NOTE THAT FOREIGN TERRITORIES OF MAIN COUNTRIES ARE
## RENAMED AS THE MAIN COUNTRY ITSELF: THIS BECAUSE WE SUPPOSE STATS
## ARE AGGREGATED AT COUNTRY LEVEL. YMMV. EVENTUALLY FIX ACCORDINGLY
##
#############################################################
# DEFINES COUNTRY NAMES FIXES DICTIONARY
countries_fixes_dict = {'CAR': 'Central African Republic',
'Congo': 'Congo (Brazzaville)',
'DRC': 'Congo (Kinshasa)',
'Diamond Princess': 'Cruise Ship',
'Ivory Coast': 'Cote d\'Ivoire',
'S. Korea': 'Korea, South',
'St. Vincent Grenadines': 'Saint Vincent and the Grenadines',
'Taiwan': 'Taiwan*',
'UAE': 'United Arab Emirates',
'UK': 'United Kingdom',
'USA': 'US',
'Vatican City': 'Holy See',
'Aruba': 'Netherlands',
'Bermuda': 'United Kingdom',
'Cayman Islands': 'United Kingdom',
'Channel Islands': 'United Kingdom',
'Curaçao': 'Netherlands',
'Faeroe Islands': 'Denmark',
'French Polynesia': 'France',
'French Guiana': 'France',
'Gibraltar': 'United Kingdom',
'Greenland': 'Denmark',
'Guadeloupe': 'France',
'Guam': 'US',
'Hong Kong': 'China',
'Isle of Man': 'United Kingdom',
'Macao': 'China',
'Martinique': 'France',
'Mayotte': 'France',
'Montserrat': 'United Kingdom',
'New Caledonia': 'France',
'Puerto Rico': 'US',
'Réunion': 'France',
'Saint Martin': 'France',
'Sint Maarten': 'Netherlands',
'St. Barth': 'France',
'Turks and Caicos': 'United Kingdom',
'U.S. Virgin Islands': 'US',
'Bahamas': 'Bahamas, The',
'Gambia': 'Gambia, The'
}
#############################################################
##
## READS WORLDOMETER STATS
##
## WEB PAGES ARE COMPLEX AND CAN CONTAIN SEVERAL TABLES. THIS IS WHY READ_HTML RETURNS
## A LIST OF TABLES. IN THIS CASE THE TABLES WE NEED ARE THE FIRST IN THE LIST
## (HENCE THE INDEXING [0]) CONTAINING THE WORLDOMETER STATS FROM TODAY, AND THE
## SECOND ONE (HENCE THE INDEXING [1]) CONTAINING THE WORLDOMETER STATS FROM YESERDAY.
## THE OPTION "header=[0]" MEANS THAT JUST THE FIRST ROW OF
## THE TABLE MAKE UP THE HEADER OTHERWISE IT COULD BE E.G. "header=[0,1]" AND IN
## THIS CASE IT WOULD MEAN THE HEADER IS MADE UP FROM THE FIRST 2 LINES OF
## THE SCRAPED TABLE. PLEASE NOTE THAT THE LAST CALL IN THE FOLLOWING CODE LINES
## IS ".iloc[:-1]" AS TO DISCARD THE LAST LINE OF THE SCRAPED TABLE: THAT WOULD
## CONTAIN GRANDTOTALS THAT WE'RE NOT INTERESTED IN
##
#############################################################
class CovidReader:
url = "https://www.worldometers.info/coronavirus/"
header = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
"X-Requested-With": "XMLHttpRequest"
}
def __init__(self):
self._now = datetime.now()
self._yesterday = date.today() - timedelta(days=1)
self._today_df = None
self._yesterday_df = None
self._recovered_df = None
def now(self):
return self._now
def yesterday(self):
return self._yesterday
def today_df(self):
return self._today_df
def yesterday_df(self):
return self._yesterday_df
def recovered_df(self):
return self._recovered_df
def fetch(self):
self._fetch_data()
self._fix_data()
# PULLS TODAY'S AND YESTERDAY'S DATA FROM WORLDOMETER SITE
def _fetch_data(self):
self._now = datetime.now()
self._yesterday = date.today() - timedelta(days=1)
r = requests.get(CovidReader.url, headers=CovidReader.header)
self._today_df = pd.read_html(r.text, header=[0], index_col=0)[0].iloc[:-1]
self._yesterday_df = pd.read_html(r.text, header=[0], index_col=0)[1].iloc[:-1]
def _fix_data(self):
# RENAMES NEW DATAFRAMES INDEX TO MATCH JHU DATAFRAME STANDARDS
self._today_df.index.names = ['Country/Region']
self._yesterday_df.index.names = ['Country/Region']
# FIXES DATATYPES AND COLUMN NAMES ON THE NEW RECOVERED STATS DATAFRAME TO MATCH JHU DATAFRAME STANDARDS
# HERE, WE JUST USE "TODAY" DATAFRAME BECAUSE WE SUPPOSE WE ARE GENERATING OUR STATS ON THE "NEW DAY"
# SO WE HAVE TO LOOK AT NEW ACTIVE VALUES SINCE YESTERDAY
self._recovered_df = self._today_df['TotalRecovered'].copy().to_frame()
self._recovered_df['TotalRecovered'] = self._recovered_df['TotalRecovered'].fillna(0).astype(np.int64)
# RENAMES NEW DATAFRAME RECOVERED COLUMN NAME TO MATCH JHU DATAFRAME STANDARDS
# USING THE "date_time_yesterday_columnname" VARIABLE WE PREVIOUSLY INITIALIZED
# REMEMBER WE WANT TO CREATE A NEW COLUMN TO HSTACK TO RECOVERED INCREMENTAL CSV
# AS YESTERDAY'S STATS/COLUMN NAME!
#
# MISMATCH IN DATE FORMAT AS DOCUMENTED IN:
# https://stackoverflow.com/questions/10807164/python-time-formatting-different-in-windows
if os.name == 'nt':
date_time_columnname = self._yesterday.strftime("%m/%d/%Y")
else:
date_time_columnname = self._yesterday.strftime("%-m/%-d/%Y")
self._recovered_df = self._recovered_df.rename(columns={'TotalRecovered': date_time_columnname})
# RENAMES COUNTRY NAMES INSIDE THE INDEX OF OUR NEW DATAFRAME TO MATCH JHU DATAFRAME STANDARDS
# USING THE FIX INDEX DICTIONARY "countries_fixes_dict" THAT WE PREVIOUSLY INITIALIZED
self._recovered_df = self._recovered_df.rename(index = countries_fixes_dict)
# AS NOW WE'LL HAVE MULTIPLE RECORDS FOR MANY COUNTRIES HAVING FOREIGN TERRITORIES (WE RENAMED
# ALL THE TERRITORIES AS THE MAIN COUNTRIES) WE GROUP AND SUM THE DATAFRAME AT COUNTRY LEVEL
# WE ALSO CHANGE NAME TO THIS DATAFRAME BECAUSE THIS IS THE GROUPED BY COUNTRY VERSION OF THE
# ORIGINAL WORLDCOUNTER DATAFRAME
def recovered_by_country(self):
df = self._recovered_df.groupby(['Country/Region']).sum()
return df
# SAVES THE "_recovered_df" DATAFRAME WITH A PROPER NAME ENRICHED WITH DATE AND TIME
# SO THAT WE HAVE A VERSION OF THE ORIGINAL DATA
def save_recovered(self, basename=r'./covid_recovered_', **kwargs):
# BUILDS TRAILER STRINGS TO APPEND TO THE FILE NAMES WE WILL USE AND TO USE TO FIX COLUMN NAMES
# IN THE NEW RECOVERIES DATAFRAME THAT WE'LL INTEGRATE IN JHU DATA
date_time_trailer = self._now.strftime("%Y_%m_%d")
self._recovered_df.to_csv(basename + date_time_trailer + '.csv', **kwargs)
Brilliant, @efra-mx !! Thanks a lot for this :)
Since 'Diamond Princess' cases are seperated from 'Japan' in Worldometers table, may I know how to group Diamond Princess cases into Japan? Thanks
Something like this would help :
df.groupby(['country/other']).sum()
On Thu, 9 Apr 2020, 16:08 Marco Cheung notifications@github.com wrote:
Since 'Diamond Princess' cases are seperated from 'Japan' in Worldometers table, may I know how to group Diamond Princess cases into Japan? Thanks
[image: image] https://user-images.githubusercontent.com/29329279/78903825-46dce000-7aae-11ea-96cd-b7719671330e.png
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CSSEGISandData/COVID-19/issues/1642#issuecomment-611547727, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C7WZXYBRP4ZKESMNKVZDRLXJFZANCNFSM4LUO3NVQ .
@efra-mx Thanks for follow-up. Can group them using this function after converting object data fields into float. However, the original column 'Continent' is missing after grouping. May I know how to add back into dataframe? Thanks!
the easiest way would be, if you also use the version of my script:
import worldmeter_stats as wms
reader = wms.CovidReader() reader.fetch() df = reader.today_df() print(df)
japan = df.loc[['Japan', 'Diamond Princess']].sum(numeric_only=True)
df.loc['Japan'] = japan df = df.drop('Diamond Princess')
print(df.loc[['Japan']])
El jue., 9 de abr. de 2020 a la(s) 17:51, Marco Cheung ( notifications@github.com) escribió:
@efra-mx https://github.com/efra-mx Thanks for follow-up. However, the result does not what I expect. I checked all column fields are now in 'object' datatype. How can I change to float/int so that it can be grouped properly?
[image: image] https://user-images.githubusercontent.com/29329279/78914184-a4782900-7abc-11ea-8d3d-5a479aa63230.png
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CSSEGISandData/COVID-19/issues/1642#issuecomment-611603098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C7W5SRAC3BFLJX6QXMDLRLXVGPANCNFSM4LUO3NVQ .
Thanks @efra-mx! That works for me and sounds straightforward to understand your code with comments!
I have this error running the code ValueError: No tables found matching pattern '.+'
How to fix that?
For Pandas + Python users, I propose the following code snippet to seamlessly integrate daily data from worldometers.info straight inside your legacy code generating stuff from recoveries stats @JHU (as we all know, this data stream is now discontinued and we are left on our own, so to say)
IMPORTANT:
OUTPUT:![Output Output](https://user-images.githubusercontent.com/33959199/77686267-f5582f80-6f9c-11ea-8edb-a7c336afa5cb.png)