[Fix] class Scenario estimate_accuracy() and fit() returns KeyError

[baluballa]

Summary

snl.estimate_accuracy(phase="last") returns: KeyError: 'Value of Fatal or Recovered was not specified with keyword arguments.'

Codes

import covsirphy as cs
# Dataset preparation
data_loader = cs.DataLoader("datasets",update_interval=24)
data_loader.read_dataframe(LoadDataFramesFromMySQL(place))
jhu_data = data_loader.jhu()
# Scenario analysis
snl = cs.Scenario(country="Norway", province=place,auto_complement=True)
snl.register(jhu_data)
snl.trend(algo="Pelt-rbf", min_size=7)
snl.estimate(cs.SIR)
snl.fit()
snl.predict()

Outputs

KeyError: 'Value of Fatal or Recovered was not specified with keyword arguments.'

snl.records(variables="all", show_figure=False).head(10).to_clipboard( index=True) yelds:

    Date    Confirmed   Infected    Fatal   Recovered   Susceptible
0   2020-06-05  107 100 0   7   59181
1   2020-06-06  107 93  0   14  59181
2   2020-06-07  107 86  0   21  59181
3   2020-06-08  107 78  0   29  59181
4   2020-06-09  107 71  0   36  59181
5   2020-06-10  107 64  0   43  59181
6   2020-06-11  108 58  0   50  59180
7   2020-06-12  108 51  0   57  59180
8   2020-06-13  108 44  0   64  59180
9   2020-06-14  108 37  0   71  59180

snl.fit() returns:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_6588/410449196.py in <module>
----> 1 snl.fit()

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\analysis\scenario.py in fit(self, oxcgrt_data, name, delay, removed_cols, metric, **kwargs)
   1323         # Create training/test dataset
   1324         tracker = self._tracker(name=name)
-> 1325         param_df = tracker.track().set_index(self.DATE)[self._model.PARAMETERS].dropna()
   1326         try:
   1327             records_df = self._data.records(main=True, extras=True).set_index(self.DATE)

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\analysis\phase_tracker.py in track(self)
    153         # Use simulated data for tracking
    154         try:
--> 155             df.update(self.simulate().set_index(self.DATE))
    156         except UnExecutedError:
    157             pass

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\analysis\phase_tracker.py in simulate(self)
    375             _ = handler.add(end, param_dict=param_dict, y0_dict=y0_dict)
    376         # Perform simulation
--> 377         sim_df = handler.simulate()
    378         sim_df[self.C] = sim_df[[self.CI, self.F, self.R]].sum(axis=1)
    379         return sim_df.loc[:, self.SUB_COLUMNS]

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\ode\ode_handler.py in simulate(self)
     95                 raise ValueError(f"{param.capitalize()} is not registered for the {phase} phase.")
     96         solver = _MultiPhaseODESolver(self._model, self._first, self._tau)
---> 97         return solver.simulate(*self._info_dict.values())
     98 
     99     def _score_tau(self, tau, data, quantile):

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\ode\ode_solver_multi.py in simulate(self, *args)
     87             # Solve the initial value problem with the ODE model
     88             solver = _ODESolver(self._model, **param_dict)
---> 89             solved_df = solver.run(step_n=step_n, **y0_dict)
     90             dataframes += [solved_df.iloc[1:]] if dataframes else [solved_df]
     91         # Combine the simulation results

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\ode\ode_solver.py in run(self, step_n, **kwargs)
     50         step_n = self._ensure_natural_int(step_n, name="number")
     51         kwargs = {param: int(value) for (param, value) in kwargs.items()}
---> 52         y0_dict = self._ensure_kwargs(self._model.VARIABLES, int, **kwargs)
     53         # Calculate population
     54         population = sum(y0_dict.values())

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\util\term.py in _ensure_kwargs(self, arg_list, value_type, **kwargs)
    543         for param in arg_list:
    544             if param not in kwargs:
--> 545                 raise KeyError(f"Value of {param} was not specified with keyword arguments.")
    546             self._ensure_instance(kwargs[param], value_type, name=f"{param} value")
    547         return {param: kwargs[param] for param in arg_list}

KeyError: 'Value of Fatal or Recovered was not specified with keyword arguments.'

and snl.estimate_accuracy(phase="last") returns:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_6588/1828239326.py in <module>
      1 # phase="last" means the last phases
----> 2 snl.estimate_accuracy(phase="last")

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\analysis\scenario.py in estimate_accuracy(self, phase, name, **kwargs)
    827         records_df = self.records(variables=variables, show_figure=False)
    828         tracker = self._tracker(name=name)
--> 829         sim_df = tracker.simulate()
    830         start, end = tracker.parse_range(phases=[phase])
    831         df = records_df.merge(sim_df, on=self.DATE, suffixes=("_actual", "_simulated"))

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\analysis\phase_tracker.py in simulate(self)
    375             _ = handler.add(end, param_dict=param_dict, y0_dict=y0_dict)
    376         # Perform simulation
--> 377         sim_df = handler.simulate()
    378         sim_df[self.C] = sim_df[[self.CI, self.F, self.R]].sum(axis=1)
    379         return sim_df.loc[:, self.SUB_COLUMNS]

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\ode\ode_handler.py in simulate(self)
     95                 raise ValueError(f"{param.capitalize()} is not registered for the {phase} phase.")
     96         solver = _MultiPhaseODESolver(self._model, self._first, self._tau)
---> 97         return solver.simulate(*self._info_dict.values())
     98 
     99     def _score_tau(self, tau, data, quantile):

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\ode\ode_solver_multi.py in simulate(self, *args)
     87             # Solve the initial value problem with the ODE model
     88             solver = _ODESolver(self._model, **param_dict)
---> 89             solved_df = solver.run(step_n=step_n, **y0_dict)
     90             dataframes += [solved_df.iloc[1:]] if dataframes else [solved_df]
     91         # Combine the simulation results

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\ode\ode_solver.py in run(self, step_n, **kwargs)
     50         step_n = self._ensure_natural_int(step_n, name="number")
     51         kwargs = {param: int(value) for (param, value) in kwargs.items()}
---> 52         y0_dict = self._ensure_kwargs(self._model.VARIABLES, int, **kwargs)
     53         # Calculate population
     54         population = sum(y0_dict.values())

~\AppData\Local\Programs\Python\Python39\lib\site-packages\covsirphy\util\term.py in _ensure_kwargs(self, arg_list, value_type, **kwargs)
    543         for param in arg_list:
    544             if param not in kwargs:
--> 545                 raise KeyError(f"Value of {param} was not specified with keyword arguments.")
    546             self._ensure_instance(kwargs[param], value_type, name=f"{param} value")
    547         return {param: kwargs[param] for param in arg_list}

KeyError: 'Value of Fatal or Recovered was not specified with keyword arguments.'

Environment

• CovsirPhy version: '2.22.1-alpha'
• Python version: 3.9.7 64bt
• Installation: pip
• System: WSL Windows Notebook on Visual Code

Note:

Its the same dataset as in issue 954 (https://github.com/lisphilar/covid19-sir/issues/954) only different is using ODE SIR instead of SIR-F. Troubled functions runs fine on ODE SIR-F.

Regards

[lisphilar]

kwargs (a dictionary) at line 51in ges\covsirphy\ode\ode_solver.py may have "Fatal" and "Recovered", but not "Fatal or Recovered". I think "Fatal or Recovered" key should be created to solve this error. I will investigate this issue this weekend.

[baluballa]

Ok, the SET has been trhough this process, in the dataloader without flaws.:

data_loader.lock(
    # Always required
    date="date", country="country", province="province",
    confirmed="confirmed", fatal="fatal", population="population",
    # Optional regarding location
    iso3="iso3",
    # Optional regarding JHUData
    recovered="recovered",
    # Optional regarding PCData
    tests="tests",
    # Optional regarding VaccineData
    product="product", vaccinations="vaccinations",
    vaccinated_once="vaccinated_once", vaccinated_full="vaccinated_full",
)

I have more or less followed the guide on covsirphy github pages "documentation", how ever data is from local government and scenario is for province.

[baluballa]

Fucntions still returns:

KeyError: 'Value of Fatal or Recovered was not specified with keyword arguments.'

in version '2.22.1-beta'

[lisphilar]

I tryed some changes with #960, but failed. The following script also shows KeyError.

import covsirphy as cs
loader = cs.DataLoader()
jhu_data = loader.jhu()
snl = cs.Scenario(country="Norway")
snl.register(jhu_data)
snl.trend()
snl.estimate(cs.SIR)
snl.summary()
snl.simulate()

snl.summary() worked, but snl.simulate() caused KeyError. Something error in calculating initial values of variables ("Fatal or Recovered", "Infected", "Susceptible”) in simulation.

[baluballa]

Here is the csv export from snl.records(variables="all"). Hope it helps... Province you can call whatever... Guess you can load it in to theScenariousing DataLoader load from csv.snl = cs.Scenario(country="Norway", province='local)`

Norway_Province.csv `

[lisphilar]

Thank you. I found a cause for the error and this may me fixied with #961. After merged, I will report to you.

[lisphilar]

961 was merged. 2.22.1-gamma could solve the problem.

[baluballa]

"Spot on". Solved in 2.22.1-gamma. Good job! Thanks!

[lisphilar]

My pleasure. I will release stable version 2.22.2 for #954 and #957 today!