Load custom contry data

[baluballa]

What we need to document?

Could you please show how to load country specific data for a province. Not sure if its a bug, but I always get this error:

KeyError: 'Expected columns were not included in the raw data with date, population, number, Confirmed, Recovered, Fatal, geo_name_idx. Date must be included.' 

after:

my_data = cs.CountryData("MycsvFile.csv")
my_data .set_variables(
    date="Date", confirmed="registrerte", fatal="death", recovered="Friske", province=''
)
my_data .cleaned().tail()

[lisphilar]

@baluballa , Thank you for using CovsirPhy!

Please update date="Date" with date="date" to solve the KeyError.

If not solved, please kindly provide the head 5 lines of the CSV file to find the cause.

[baluballa]

Right, thanks for quick and good answer. That got me one step further, as date column was with lover D. However still struggling and cannot find docs for it. Since this is about a province, population needs to be altered. I am not sure how to do this. When:

nor_data = cs.CountryData("D:\\Users\\XXX\\Source\\Repos\\SIR\\input\\Oslo.csv", country="Norway")
population_data = data_loader.population()
snl = cs.Scenario(nor_data,population_data, country="Norway",province='Oslo')
nor_data.set_variables(
    date="date", confirmed="confirmed", fatal="death", recovered="recovered", province='province'
)

I get SubsetNotFoundError: Records in Oslo, Norway were not found.

So where do I put the population_data , or load a custom population_data?

Here is 5 first line of CSV, which is extracted from MySQL

date,population,confirmed,infected,recovered,death,province
2020-06-11,693494,2738,11,0,0,Oslo
2020-06-11,693494,2733,11,0,0,Oslo
2020-06-12,693494,2748,10,16,0,Oslo
2020-06-12,693494,2743,10,16,0,Oslo
2020-06-15,693494,2764,16,30,0,Oslo

Hopefully you can get me a step further again or direct me to a "how to" or like?

Regards.

[lisphilar]

Thank you for the detailed information.

Please try the following codes. (version 2.12.0)

# Create country data (only with Norway/Oslo)
nor_data = cs.CountryData("D:\\Users\\XXX\\Source\\Repos\\SIR\\input\\Oslo.csv", country="Norway")
nor_data.set_variables(
    date="date", confirmed="confirmed", fatal="death", recovered="recovered", province="province"
)
# Create JHUData instance with Norway/Oslo dataset
jhu_data = cs.JHUData.from_dataframe(nor_data.cleaned())
# Create PopulationData instance with specified value
population_data = cs.PopulationData()
population_data.update(693494, country="Norway", province="Oslo")
# Start scenario analysis
snl = cs.Scenario(jhu_data,population_data, country="Norway",province="Oslo")

• It is required to create JHUData with CountryData.cleaned() (pandas.DataFrame) because CountryData class is not a child class of JHUData.
• We need to register the population value to PopulationData instance with .update() method.

As you mentioned, this should be documented (and tested).

If this script is difficult to use, we may create a new class LocalData for handling local files. Do you have any ideas for convenient usage of this interface? E.g. local_data = cs.LocalData(country="Norway", province="Oslo"); jhu_data = local_data(confirmed="confirmed");...

[baluballa]

So I have pasted your siggestions into my notebook. The last commandsnl = cs.Scenario(jhu_data,population_data, country="Norway",province="Oslo") gives this error:

C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\pandas\core\reshape\reshape.py in stack(frame, level, dropna)
    520         # we concatenate instead.
    521         dtypes = list(frame.dtypes._values)
--> 522         dtype = dtypes[0]
    523 
    524         if is_extension_array_dtype(dtype):

IndexError: list index out of range

last trace lines...

Version of python is Python 3.7.8 and 'CovsirPhy v2.12.0-rho.new.339.357.359.362.366.367.383.fix.360.373'

[lisphilar]

With my local file dataset (Japan), this script worked in verison 2.12.0, but not worked in version 2.12.0-rho as you mentioned. I will try to find the cause before releasing the next stable version!

Memo: Minimal codes to raise the same error (country_data is a instance of CountryData) is as follows.

jhu_data = cs.JHUData.from_dataframe(country_data.cleaned())
# Not raise
jhu_data.subset("country name")
# Raise IndexError
jhu_data.subset_complement("country name")

The root cause is that JHUData.calculate_recovery_period() does not avoid pd.DataFrame().stack() when the datafame is empty.

[lisphilar]

With #408, I fixed the IndexError. Kindly confirm this with your dataset.

[baluballa]

Tested 'CovsirPhy v2.12.0-phi.new.339.357.359.362.366.367.382.383.389.408.fix.360.373.406' with my custom dataset which is described above. I can confirm it works perfect with the code as above.

Thanks!

[lisphilar]

Thank you for confirmation! I updated documenation and this will be shown in several hours. https://lisphilar.github.io/covid19-sir/

I will close this issues, but you can reopen this issue or create new issues, if necessary.

[lisphilar]

Dear @baluballa , This is just an follow-up message. If you have any issues of CovisirPhy project, kindly let me know. Thank you for your analysis!