RossBoylan commented 4 years ago

pytest.log has the full results of running the test suite. It appears all tests passed, but most had warnings.

The warnings are mostly of two types: bad numbers, and bad optimization options. So I'll put one example of each.

Numerical issues

Lots of complaints about invalid values, some of them accompanied by division by zero warnings.

tests/adapter_covid19/test_corporate_bankruptcy_models.py::TestClass::()::test_interface[CorporateBankruptcyModel-utilisation0-final_use_shortfall_vs_demand0-primary_inputs0]
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:359: RuntimeWarning: invalid value encountered in double_scalars
    [days / total_solvent_days * cash_buffer for days in solvent_days]
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:301: RuntimeWarning: invalid value encountered in greater
    s: self.init_cash_state[BusinessSize.large][s] > 0 for s in Sector
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:621: RuntimeWarning: divide by zero encountered in true_divide
    for s in Sector
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:621: RuntimeWarning: invalid value encountered in multiply
    for s in Sector
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:654: RuntimeWarning: invalid value encountered in greater
    self.cash_state[BusinessSize.large][s] > 0
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:262: RuntimeWarning: invalid value encountered in true_divide
    return vector / np.sum(vector)
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:413: RuntimeWarning: invalid value encountered in greater
    solvent = float(np.mean(cash_buffer_sample > 0))

Optimization Options

Every example I noticed was about the same option:

tests/adapter_covid19/test_economics.py::TestClass::()::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-NaiveCorporateBankruptcyModel-utilisation3]
  /home/ross/.local/lib/python3.7/site-packages/adapter_covid19/gdp.py:999: OptimizeWarning: Unknown solver options: autoscale
    options={"maxiter": 1e4, "disp": False, "autoscale": False},
  /usr/lib/python3/dist-packages/scipy/optimize/_linprog_ip.py:643: OptimizeWarning: A_eq does not appear to be of full row rank. To improve performance, check the problem formulation for redundant equality constraints.
    warn(redundancy_warning, OptimizeWarning)

That one has a bonus numerical complaint, but others have just the complaint about autoscale.

Environment

Running on Debian 10=buster. gcc 8.3.0-6 python 3.7.3 swig 3.0.12-3

RossBoylan commented 4 years ago

On the numerical problems, I note that the Makefile builds the C modules with -O0 while the link options specify -O3. Perhaps that's causing trouble?

p-robot commented 4 years ago

Thanks. This has been flagged with the economics team. From my understanding these are confined to the module adapter_covid19 (the economics module) and there are no warnings thrown from the tests of the epidemiological model, the interventions, or of concordance of the output files.

RossBoylan commented 4 years ago

Building from the more recent master, 81768675c8ca43836f1143f0e0af7504509bdc2b, which appears to be a much different development branch than a few days ago, I now get one failure and a shorter list of warnings. I did build with compiler option -O3 and with -g this time; not sure if that has anything to do with the shorter list of warnings. This test was also executed in a python venv.

See at the end for notes on peculiarities in the reported warnings.

(jarenv) ross@barley:/usr/local/jar/OpenABM-Covid19$ date; time python3 -m pytest
Sun 14 Jun 2020 11:28:40 AM PDT
======================================================== test session starts =========================================================
platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /usr/local/jar/OpenABM-Covid19
collected 189 items                                                                                                                  

tests/test_demographics.py .........                                                                                           [  4%]
tests/test_disease_dynamics.py ................                                                                                [ 13%]
tests/test_file_concordance.py ........                                                                                        [ 17%]
tests/test_ibm.py ..........                                                                                                   [ 22%]
tests/test_infection_dynamics.py ........................F..                                                                   [ 37%]
tests/test_interventions.py .......................                                                                            [ 49%]
tests/test_network.py ............................                                                                             [ 64%]
tests/test_python_c_interface.py .....                                                                                         [ 66%]
tests/test_python_param_setting.py ...                                                                                         [ 68%]
tests/test_set_parameters.py ......                                                                                            [ 71%]
tests/adapter_covid19/test_corporate_bankruptcy_models.py ............                                                         [ 77%]
tests/adapter_covid19/test_data_structures.py .                                                                                [ 78%]
tests/adapter_covid19/test_economics.py ........                                                                               [ 82%]
tests/adapter_covid19/test_gdp_models.py .....                                                                                 [ 85%]
tests/adapter_covid19/test_personal_bankruptcy_models.py ....                                                                  [ 87%]
tests/hospital/test_hospital_logic.py .......                                                                                  [ 91%]
tests/hospital/test_hospital_networks.py ...                                                                                   [ 92%]
tests/hospital/test_parameter_boundaries.py ..............                                                                     [100%]

============================================================== FAILURES ==============================================================
___________________________________ TestClass.test_ratio_presymptomatic_symptomatic[1-100-1000000] ___________________________________

self = <tests.test_infection_dynamics.TestClass object at 0x7f171e36fd68>, n_total = 1000000, n_seed_infection = 100, end_time = 1

    def test_ratio_presymptomatic_symptomatic(
            self,
            n_total,
            n_seed_infection,
            end_time
        ):
        """
        Test that ratio presymptomatic to symptomatic individuals is correct; currently must be 1.
        """
        tolerance = 1/n_total

        params = ParameterSet(constant.TEST_DATA_FILE, line_number=1)

        params.set_param("n_total", n_total)
        params.set_param("n_seed_infection", n_seed_infection)
        params.set_param("end_time", end_time)
        params.write_params(constant.TEST_DATA_FILE)
        file_output = open(constant.TEST_OUTPUT_FILE, "w")
        completed_run = subprocess.run([constant.command], stdout=file_output, shell=True)
        df_indiv = pd.read_csv(
            constant.TEST_INDIVIDUAL_FILE, comment="#", sep=",", skipinitialspace=True
        )

>       df_trans = pd.read_csv(constant.TEST_TRANSMISSION_FILE)

tests/test_infection_dynamics.py:889: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:676: in parser_f
    return _read(filepath_or_buffer, kwds)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:448: in _read
    parser = TextFileReader(fp_or_buf, **kwds)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:880: in __init__
    self._make_engine(self.engine)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:1114: in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:1891: in __init__
    self._reader = parsers.TextReader(src, **kwds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pandas.errors.EmptyDataError: No columns to parse from file

pandas/_libs/parsers.pyx:532: EmptyDataError
========================================================== warnings summary ==========================================================
tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
tests/adapter_covid19/test_economics.py: 4 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:359: RuntimeWarning: invalid value encountered in double_scalars
    [days / total_solvent_days * cash_buffer for days in solvent_days]

tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
tests/adapter_covid19/test_economics.py: 4 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:301: RuntimeWarning: invalid value encountered in greater
    s: self.init_cash_state[BusinessSize.large][s] > 0 for s in Sector

tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:621: RuntimeWarning: divide by zero encountered in true_divide
    for s in Sector

tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
tests/adapter_covid19/test_economics.py: 4 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:621: RuntimeWarning: invalid value encountered in multiply
    for s in Sector

tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
tests/adapter_covid19/test_economics.py: 4 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:654: RuntimeWarning: invalid value encountered in greater
    self.cash_state[BusinessSize.large][s] > 0

tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
tests/adapter_covid19/test_economics.py: 4 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:262: RuntimeWarning: invalid value encountered in true_divide
    return vector / np.sum(vector)

tests/adapter_covid19/test_corporate_bankruptcy_models.py: 12 tests with warnings
tests/adapter_covid19/test_economics.py: 4 tests with warnings
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:413: RuntimeWarning: invalid value encountered in greater
    solvent = float(np.mean(cash_buffer_sample > 0))

tests/adapter_covid19/test_economics.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-NaiveCorporateBankruptcyModel-utilisation3]
tests/adapter_covid19/test_economics.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-CorporateBankruptcyModel-utilisation3]
tests/adapter_covid19/test_gdp_models.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-utilisation3]
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/gdp.py:999: OptimizeWarning: A_eq does not appear to be of full row rank. To improve performance, check the problem formulation for redundant equality constraints.
    options={"maxiter": 1e4, "disp": False, "autoscale": False},

tests/adapter_covid19/test_economics.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-CorporateBankruptcyModel-utilisation0]
tests/adapter_covid19/test_economics.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-CorporateBankruptcyModel-utilisation1]
tests/adapter_covid19/test_economics.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-CorporateBankruptcyModel-utilisation2]
tests/adapter_covid19/test_economics.py::TestClass::test_interface[PiecewiseLinearCobbDouglasGdpModel-PersonalBankruptcyModel-CorporateBankruptcyModel-utilisation3]
  /usr/local/jar/jarenv/lib/python3.7/site-packages/adapter_covid19/corporate_bankruptcy.py:621: RuntimeWarning: divide by zero encountered in double_scalars
    for s in Sector

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================================================== short test summary info =======================================================
FAILED tests/test_infection_dynamics.py::TestClass::test_ratio_presymptomatic_symptomatic[1-100-1000000] - pandas.errors.EmptyDataE...
====================================== 1 failed, 188 passed, 115 warnings in 419.03s (0:06:59) =======================================

real    6m59.466s
user    6m58.661s
sys 0m29.132s

The test reports 115 warnings, but far fewer seemed to be reported above it. The entire log is shown above. The statement in my first post that most tests had warnings was based on the assumption that 128 warnings reported there meant 128 tests had warnings, but that was clearly not the case in the earlier run. I thought there might have been 128 warnings in the earlier run, but there are clearly not 115 reported warnings in this one.

RossBoylan commented 4 years ago

After merging in https://github.com/al626/OpenABM-Covid19/tree/arun--economics-warnings, all the warnings are gone. The failure remains.

(jarenv) ross@barley:/usr/local/jar/OpenABM-Covid19$ date; time python3 -m pytest
Sun 14 Jun 2020 12:09:35 PM PDT
======================================================== test session starts =========================================================
platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /usr/local/jar/OpenABM-Covid19
collected 189 items                                                                                                                  

tests/test_demographics.py .........                                                                                           [  4%]
tests/test_disease_dynamics.py ................                                                                                [ 13%]
tests/test_file_concordance.py ........                                                                                        [ 17%]
tests/test_ibm.py ..........                                                                                                   [ 22%]
tests/test_infection_dynamics.py ........................F..                                                                   [ 37%]
tests/test_interventions.py .......................                                                                            [ 49%]
tests/test_network.py ............................                                                                             [ 64%]
tests/test_python_c_interface.py .....                                                                                         [ 66%]
tests/test_python_param_setting.py ...                                                                                         [ 68%]
tests/test_set_parameters.py ......                                                                                            [ 71%]
tests/adapter_covid19/test_corporate_bankruptcy_models.py ............                                                         [ 77%]
tests/adapter_covid19/test_data_structures.py .                                                                                [ 78%]
tests/adapter_covid19/test_economics.py ........                                                                               [ 82%]
tests/adapter_covid19/test_gdp_models.py .....                                                                                 [ 85%]
tests/adapter_covid19/test_personal_bankruptcy_models.py ....                                                                  [ 87%]
tests/hospital/test_hospital_logic.py .......                                                                                  [ 91%]
tests/hospital/test_hospital_networks.py ...                                                                                   [ 92%]
tests/hospital/test_parameter_boundaries.py ..............                                                                     [100%]

============================================================== FAILURES ==============================================================
___________________________________ TestClass.test_ratio_presymptomatic_symptomatic[1-100-1000000] ___________________________________

self = <tests.test_infection_dynamics.TestClass object at 0x7f85ef95c2b0>, n_total = 1000000, n_seed_infection = 100, end_time = 1

    def test_ratio_presymptomatic_symptomatic(
            self,
            n_total,
            n_seed_infection,
            end_time
        ):
        """
        Test that ratio presymptomatic to symptomatic individuals is correct; currently must be 1.
        """
        tolerance = 1/n_total

        params = ParameterSet(constant.TEST_DATA_FILE, line_number=1)

        params.set_param("n_total", n_total)
        params.set_param("n_seed_infection", n_seed_infection)
        params.set_param("end_time", end_time)
        params.write_params(constant.TEST_DATA_FILE)
        file_output = open(constant.TEST_OUTPUT_FILE, "w")
        completed_run = subprocess.run([constant.command], stdout=file_output, shell=True)
        df_indiv = pd.read_csv(
            constant.TEST_INDIVIDUAL_FILE, comment="#", sep=",", skipinitialspace=True
        )

>       df_trans = pd.read_csv(constant.TEST_TRANSMISSION_FILE)

tests/test_infection_dynamics.py:889: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:676: in parser_f
    return _read(filepath_or_buffer, kwds)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:448: in _read
    parser = TextFileReader(fp_or_buf, **kwds)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:880: in __init__
    self._make_engine(self.engine)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:1114: in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
../jarenv/lib/python3.7/site-packages/pandas/io/parsers.py:1891: in __init__
    self._reader = parsers.TextReader(src, **kwds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pandas.errors.EmptyDataError: No columns to parse from file

pandas/_libs/parsers.pyx:532: EmptyDataError
====================================================== short test summary info =======================================================
FAILED tests/test_infection_dynamics.py::TestClass::test_ratio_presymptomatic_symptomatic[1-100-1000000] - pandas.errors.EmptyDataE...
============================================= 1 failed, 188 passed in 408.10s (0:06:48) ==============================================

real    6m48.326s
user    6m50.872s
sys 0m26.888s

p-robot commented 4 years ago

Thanks for flagging. PR for #148 has no failing tests and only 3 warnings (which are SettingWithCopyWarning warnings from pandas; here and pasted below). PRs are typically only merged once all tests have passed.

============================= test session starts ==============================
platform linux -- Python 3.7.8, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /OpenABM-Covid19
collected 246 items

tests/test_demographics.py ..........
tests/test_disease_dynamics.py ................
tests/test_file_concordance.py .........
tests/test_ibm.py ..........
tests/test_infection_dynamics.py ...............................
tests/test_interventions.py ........................................
tests/test_network.py ........................................
tests/test_python_c_interface.py .....
tests/test_python_param_setting.py ...
tests/test_set_parameters.py .......
tests/test_utilities.py .....................
tests/adapter_covid19/test_corporate_bankruptcy_models.py ............
tests/adapter_covid19/test_data_structures.py .
tests/adapter_covid19/test_economics.py ........
tests/adapter_covid19/test_gdp_models.py .....
tests/adapter_covid19/test_personal_bankruptcy_models.py ....
tests/hospital/test_hospital_logic.py .......
tests/hospital/test_hospital_networks.py ...
tests/hospital/test_parameter_boundaries.py ..............

=============================== warnings summary ===============================
tests/test_interventions.py::TestClass::test_priority_testing[1.0-30-test_params0]
tests/test_interventions.py::TestClass::test_priority_testing[1.0-30-test_params1]
tests/test_interventions.py::TestClass::test_priority_testing[1.0-30-test_params2]
  /OpenABM-Covid19/tests/test_interventions.py:1756: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead

  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    test_df_symp[ "priority_symp" ] = ( test_df_trans["time_symptomatic"] ==  time_prior_symp )

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================= 246 passed, 3 warnings in 1556.42s (0:25:56) =================

BDI-pathogens / OpenABM-Covid19

test suite warnings #87

Numerical issues

Optimization Options

Environment