IAMconsortium / pyam

Analysis & visualization of energy & climate scenarios
https://pyam-iamc.readthedocs.io/
Apache License 2.0
221 stars 115 forks source link

Concatenating non-pyam dataframes no longer works #692

Open Rlamboll opened 1 year ago

Rlamboll commented 1 year ago

Previously, one could pyam-concatenate pandas dataframes that can be cast to pyam format directly. However this now fails as the meta looks for a metadata index that isn't present. I assume this is an unintended bug rather than a safety feature, if not the documentation should be updated to make clear that concatenation only works on the pyam format itself.

The problem:

return IamDataFrame(
            pd.concat(ret_data, verify_integrity=False),
            meta=ret_meta,
>           index=ret_meta.index.names,
        )
E       AttributeError: 'NoneType' object has no attribute 'index'
gidden commented 1 year ago

I also noticed this but never got around to making an issue. Thanks @Rlamboll ! I agree, this felt like a regression to me..

danielhuppmann commented 1 year ago

I was going to write that we don't have a test for pd.DataFrame-only-concatenation, but I can't find a test for concat() at all... ?

Anyway, I think it should be sufficient to replace

https://github.com/IAMconsortium/pyam/blob/e7f0c3a2e8ba618e1b621333064657e38db898c4/pyam/core.py#L2935

with

index = df.index.names

and write a test for it...