concat not working with non-standard index dimensions

[phackstock]

When concatenating IamDataFrame objects that have non-standard index dimensions (i.e. anything other than model, scenario) you currently get an error merging the meta indicators.

Minimum example to reproduce

Creating two IamDataFrames df1, df2 with index dimensions model, scenario, version and running pyam.concat:

import pyam
import pandas as pd

df1 = pyam.IamDataFrame(
    pd.DataFrame(
        [["model_a", "scenario_a", 1, "region_a", "variable_a", "unit", 1]],
        columns=["model", "scenario", "version", "region", "variable", "unit", 2005],
    ),
    index=pyam.DEFAULT_META_INDEX + ["version"],
)
df2 = pyam.IamDataFrame(
    pd.DataFrame(
        [["model_a", "scenario_a", 2, "region_a", "variable_a", "unit", 2]],
        columns=["model", "scenario", "version", "region", "variable", "unit", 2005],
    ),
    index=pyam.DEFAULT_META_INDEX + ["version"],
)

pyam.concat([df1, df2])

yields the following error:

ValueError: Incompatible `index=['model', 'scenario']` with `meta` (index=['model', 'scenario', 'version'])!

Proposed fix

As the last step of pyam.concat is the creation of a new IamDataFrame it should work so simply carry over the index dimensions of the concatenated meta table, like so:

return IamDataFrame(
        pd.concat(ret_data, verify_integrity=False), meta=ret_meta, index = ret_meta.index.names
    )

[danielhuppmann]

Thanks for looking into this issue! Sweet that the fix is so simple...

[phackstock]

Yes, looks like the fix was simple in the end. Hopefully it does not have any unintended side-effects but none of the tests broke and I added an explicit unit test in the PR.