iiasa / climate-assessment

https://climate-assessment.readthedocs.io/en/latest
MIT License
19 stars 18 forks source link

Harmonization and Infilling fails if only `Emissions|CO2` is provided but no `Emissions|CO2|Energy and Industrial Processes` #20

Open jkikstra opened 1 year ago

jkikstra commented 1 year ago

This a bug and/or something to probably mention in the docs.

Desired behaviour It should be possible to perform a climate run with either Emissions|CO2 or Emissions|CO2|Energy and Industrial Processes reported, at minimum.

Issue With the input ar6_minimum_emissions_co2.csv, after resolving the issue described in issue #19 , we get a bit further in the processing chain, but at the end of harmonisation_and_infilling we get an error looking like what is below (running run-example-fair.ipynb).

This error is thrown by the sanity_check_hierarchy(), which has co2_infill_db as input - so that's where the investigation should start.

...

2022-12-12 14:40:47 climate_assessment.infilling MainThread - INFO:  Post-processing for climate models
2022-12-12 14:40:47 climate_assessment.infilling MainThread - INFO:  Checking infilled results have required years and variables
2022-12-12 14:40:47 climate_assessment.infilling MainThread - INFO:  Check that there are no non-CO2 negatives introduced by infilling
2022-12-12 14:40:47 climate_assessment.harmonization_and_infilling MainThread - INFO:  Writing infilled data as csv to: ..\data\output-fair-example-notebook\ar6_minimum_emissions_co2_harmonized_infilled.csv
2022-12-12 14:40:47 climate_assessment.harmonization_and_infilling MainThread - INFO:  Writing infilled data as xlsx to: ..\data\output-fair-example-notebook\ar6_minimum_emissions_co2_harmonized_infilled.xlsx
2022-12-12 14:40:47 pyam.utils MainThread - WARNING:  Formatted data is empty!

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_2548\3282750836.py in <module>
      8     num_cfgs=num_cfgs,
      9     infilling_database=infilling_database_file,
---> 10     scenario_batch_size=scenario_batch_size,
     11 )

...

c:\users\kikstra\documents\github\climate-assessment\src\climate_assessment\harmonization_and_infilling.py in harmonization_and_infilling(df, key_string, infilling_database, prefix, instance, outdir, do_harmonization)
    133                 infilled,
    134                 out_afolu="Emissions|CO2|AFOLU",
--> 135                 out_fossil="Emissions|CO2|Energy and Industrial Processes",
    136             )
    137 

c:\users\kikstra\documents\github\climate-assessment\src\climate_assessment\checks.py in sanity_check_hierarchy(co2_inf_db, harmonized, infilled, out_afolu, out_fossil)
   1101     )
   1102 
-> 1103     if not (np.isclose(infill_db_pivot, harmonized_pivot)).all():
   1104         raise ValueError(
   1105             "The sum of AFOLU and Energy and Industrial Processes "

<__array_function__ internals> in isclose(*args, **kwargs)

~\.conda\envs\ca-testing\lib\site-packages\numpy\core\numeric.py in isclose(a, b, rtol, atol, equal_nan)
   2356     yfin = isfinite(y)
   2357     if all(xfin) and all(yfin):
-> 2358         return within_tol(x, y, atol, rtol)
   2359     else:
   2360         finite = xfin & yfin

~\.conda\envs\ca-testing\lib\site-packages\numpy\core\numeric.py in within_tol(x, y, atol, rtol)
   2337     def within_tol(x, y, atol, rtol):
   2338         with errstate(invalid='ignore'):
-> 2339             return less_equal(abs(x-y), atol + rtol * abs(y))
   2340 
   2341     x = asanyarray(a)

ValueError: operands could not be broadcast together with shapes (86,1) (0,1) 

Proposed minimum solution tbd. First check how sanity_check_hierarchy() is implemented, and figure out why it throws the error.

Proposed desired solution tbd.