search

openscm / pymagicc

Python wrapper for the simple climate model MAGICC
https://pymagicc.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
40 stars 27 forks source link

Missing aerosols and gasses in pymagicc.run output #243

Closed safiume closed 3 years ago

safiume commented 5 years ago

Describe the bug

Want to able to show the output of the Montreal Protocol Gasses, and any other FGasses and Halogasses, as well as organic carbon and Ammonia. live.magicc.org supports the ability to individually graph all the aerosols as well as other forcing organics. To work around this, I manually run MAGICC and process DATBASKET*.OUT into a gsheet and can graph them as needed.

Related to #86, #107, #181

Failing Test

N/A

Expected behavior

I only see the following returned from pymagicc.run: image

Would like to get all the gasses: e.g., aerosols, and organics similar to the output listed in the tab ONClmin.data_rf in the spreadsheet https://docs.google.com/spreadsheets/d/1MpIExFIHg8dZyzXVK6c0uHdemJuXQVoGHL3RdXoiWAg

To work around this, I converted the DATBASKET.OUT after line 22 to a delineated file format and then added them the above gsheet and also generated a Radiative Forcing graph (see the data_rf, data_mid, *data_emissions tabs).

Screenshots

Desired similar to the following: image

System :

Additional context

rgieseke commented 5 years ago

I think you need to set the out_* flags to get additional output:

out = pymagicc.run(rcp26, out_forcing=1)

This should give you these variables

'Radiative Forcing', 'Radiative Forcing|Aerosols|Direct Effect',
       'Radiative Forcing|Aerosols|Indirect Effect',
       'Radiative Forcing|Aerosols|MAGICC AFOLU',
       'Radiative Forcing|Anthropogenic',
       'Radiative Forcing|BC|MAGICC AFOLU',
       'Radiative Forcing|BC|MAGICC Fossil and Industrial',
       'Radiative Forcing|Black Carbon on Snow', 'Radiative Forcing|C2F6',
       'Radiative Forcing|C6F14', 'Radiative Forcing|CCl4',
       'Radiative Forcing|CF4', 'Radiative Forcing|CFC11',
       'Radiative Forcing|CFC113', 'Radiative Forcing|CFC114',
       'Radiative Forcing|CFC115', 'Radiative Forcing|CFC12',
       'Radiative Forcing|CH3Br', 'Radiative Forcing|CH3CCl3',
       'Radiative Forcing|CH3Cl', 'Radiative Forcing|CH4',
       'Radiative Forcing|CH4 Oxidation Stratospheric H2O',
       'Radiative Forcing|CO2', 'Radiative Forcing|CO2, CH4 and N2O',
       'Radiative Forcing|Extra', 'Radiative Forcing|F Gases',
       'Radiative Forcing|Greenhouse Gases',
       'Radiative Forcing|Greenhouse Gases|Kyoto Gases',
       'Radiative Forcing|HCFC141b', 'Radiative Forcing|HCFC142b',
       'Radiative Forcing|HCFC22', 'Radiative Forcing|HFC125',
       'Radiative Forcing|HFC134a', 'Radiative Forcing|HFC143a',
       'Radiative Forcing|HFC227ea', 'Radiative Forcing|HFC23',
       'Radiative Forcing|HFC245fa', 'Radiative Forcing|HFC32',
       'Radiative Forcing|HFC4310', 'Radiative Forcing|Halon1202',
       'Radiative Forcing|Halon1211', 'Radiative Forcing|Halon1301',
       'Radiative Forcing|Halon2402', 'Radiative Forcing|Land-use Change',
       'Radiative Forcing|Mineral Dust',
       'Radiative Forcing|Montreal Protocol Halogen Gases',
       'Radiative Forcing|N2O', 'Radiative Forcing|NOx|MAGICC AFOLU',
       'Radiative Forcing|NOx|MAGICC Fossil and Industrial',
       'Radiative Forcing|OC|MAGICC AFOLU',
       'Radiative Forcing|OC|MAGICC Fossil and Industrial',
       'Radiative Forcing|SF6', 'Radiative Forcing|SOx|MAGICC AFOLU',
       'Radiative Forcing|SOx|MAGICC Fossil and Industrial',
       'Radiative Forcing|Solar', 'Radiative Forcing|Stratospheric Ozone',
       'Radiative Forcing|Tropospheric Ozone',
       'Radiative Forcing|Volcanic',
safiume commented 5 years ago

Hi, I tried that, and it errors. Should I pull the latest, and try again? I'm running pymagicc version % 2.0.0-alpha.1+156.g9297496.dirty

%time results = pymagicc.run(rcp26, out_forcing=1)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<timed exec> in <module>

~/workbench/pymagicc/pymagicc/__init__.py in run(scenario, magicc_version, **kwargs)
     73 
     74     with magicc_cls() as magicc:
---> 75         results = magicc.run(scenario=scenario, **kwargs)
     76 
     77     return results

~/workbench/pymagicc/pymagicc/core.py in run(self, scenario, only, **kwargs)
    255                         join(self.out_dir, filepath), columns=deepcopy(read_cols)
    256                     )
--> 257                     mdata = mdata.append(tempdata) if mdata is not None else tempdata
    258 
    259             except NoReaderWriterError:

~/workbench/pymagicc/pymagicc/io.py in append(self, other, inplace, **kwargs)
   1994             self.metadata.update(other.metadata)
   1995         else:
-> 1996             res = super().append(other, inplace=inplace)
   1997             res.metadata = deepcopy(self.metadata)
   1998             res.metadata.update(other.metadata)

~/workbench/pymagicc/venv/lib/python3.6/site-packages/openscm/scmdataframebase.py in append(self, other, inplace, **kwargs)
    576             other = self.__class__(other, **kwargs)
    577 
--> 578         return df_append([self, other], inplace=inplace)
    579 
    580     def set_meta(self, meta, name=None, index=None):

~/workbench/pymagicc/venv/lib/python3.6/site-packages/openscm/scmdataframebase.py in df_append(dfs, inplace)
    198     # we want to put data into timeseries format and pass into format_ts instead of format_data
    199     data = pd.concat(
--> 200         [d.timeseries().reorder_levels(joint_meta) for d in joint_dfs], sort=False
    201     )
    202 

~/workbench/pymagicc/venv/lib/python3.6/site-packages/pandas/core/reshape/concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    227                        verify_integrity=verify_integrity,
    228                        copy=copy, sort=sort)
--> 229     return op.get_result()
    230 
    231 

~/workbench/pymagicc/venv/lib/python3.6/site-packages/pandas/core/reshape/concat.py in get_result(self)
    418                     obj_labels = mgr.axes[ax]
    419                     if not new_labels.equals(obj_labels):
--> 420                         indexers[ax] = obj_labels.reindex(new_labels)[1]
    421 
    422                 mgrs_indexers.append((obj._data, indexers))

~/workbench/pymagicc/venv/lib/python3.6/site-packages/pandas/core/indexes/base.py in reindex(self, target, method, level, limit, tolerance)
   3138                         raise ValueError("cannot reindex a non-unique index "
   3139                                          "with a method or limit")
-> 3140                     indexer, missing = self.get_indexer_non_unique(target)
   3141 
   3142         if preserve_names and target.nlevels == 1 and target.name != self.name:

~/workbench/pymagicc/venv/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_indexer_non_unique(self, target)
   4437 
   4438         if self.is_all_dates:
-> 4439             self = Index(self.asi8)
   4440             tgt_values = target.asi8
   4441         else:

~/workbench/pymagicc/venv/lib/python3.6/site-packages/pandas/core/indexes/base.py in __new__(cls, data, dtype, copy, name, fastpath, tupleize_cols, **kwargs)
    466                          **kwargs)
    467         elif data is None or is_scalar(data):
--> 468             cls._scalar_data_error(data)
    469         else:
    470             if tupleize_cols and is_list_like(data):

~/workbench/pymagicc/venv/lib/python3.6/site-packages/pandas/core/indexes/base.py in _scalar_data_error(cls, data)
   3797         raise TypeError('{0}(...) must be called with a collection of some '
   3798                         'kind, {1} was passed'.format(cls.__name__,
-> 3799                                                       repr(data)))
   3800 
   3801     @classmethod

TypeError: Index(...) must be called with a collection of some kind, None was passed

Also, I'm not seeing CH4 alone, only 'Radiative Forcing|CH3Cl', 'Radiative Forcing|CH4'

And I'm not seeing emissions data for the other gasses beyond CO2 and BC, which includes NH3 among others.

rgieseke commented 5 years ago

Pymagicc development has been in a flux under master and i think the latest merge might have broken 3.6 compatibility. Can you try with Python 3.7?

rgieseke commented 5 years ago

As for the other gases, what are you looking for? This is the list of out_* params:

https://github.com/openclimatedata/pymagicc/blob/master/pymagicc/MAGICC6/run/MAGCFG_USER.CFG#L186

safiume commented 5 years ago

Ok, I'll refresh my entire kit and try again. This will have to wait till late tonight. Yes, looks like I need to specify out_forcing, out_emissions, out_concentrations. Thanks!

safiume commented 5 years ago

It seems two work but now it's quite slow... I won't be able to pick this up till much later in the week.

pymagicc version % 2.0.0-alpha.1+180.ge1103b4
python version % 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)]

##Start of Magic settings
magcfg = { "startyear" : 1765,
          "endyear" : 3000,
          "core_climatesensitivity" : 3,
          "co2_tempfeedback_yrstart" : 2005,
          "co2_fertilization_yrstart" : 2005,
          "co2_switchfromconc2emis_year" : 2005,
          "rf_mhalo_constantafteryr" : 2500,
          "rf_fgas_constantafteryr" : 2500,
          "out_forcing" : 1,
          "out_emissions" : 1,
          "out_concentrations" : 1
             }
### End MAGICC Settings

%time results = pymagicc.run(rcp26, **magcfg)
slice = results[["scenario", "region", "variable", "unit"]]
results = ""
print(slice[slice["region"].isin(["World"])])
CPU times: user 16.6 s, sys: 2.28 s, total: 18.9 s
Wall time: 22.5 s
    scenario region                                           variable  \
0      RCP26  World                                      AIR_CIRRUS_RF   
5      RCP26  World                                    AIR_CONTRAIL_RF   
10     RCP26  World                                         AIR_H2O_RF   
15     RCP26  World                    Atmospheric Concentrations|C2F6   
...

![image](https://user-images.githubusercontent.com/4840719/57567091-0306c580-7389-11e9-8d82-ec316265ba33.png)
rgieseke commented 5 years ago

Great to hear, usually that long run times should only happen if Wine is configuring for the first time (or after a re-install), and subsequent runs should take a few seconds at most.

znicholls commented 4 years ago

@safiume can we close this? The performance issues will be fixed with MAGICC7 and the functionality seems to be there even if it is slow?

safiume commented 3 years ago

Yep, go ahead and close this one. Sorry I missed this notification.