add_par() method raises an error if an empty dataframe is passed

[behnam-zakeri]

It seems the behaviour of add_par() has changed recently, not accepting an empty pandas DataFrame as input argument, raising an error. Previously it wasn't like this. This makes some of my current workflows breaking, where there is a possibility that an empty DataFrame being passed to add_par(). So, I have to find each and every add_par() in multiple scripts and add an if statement before them to avoid this error.

if not df.empty:
    scen.add_par(parname, df)

Was this change of behaviour intended? If there are good grounds for that, I'll start to modify my scripts. But if not, it will be nice to revert back to the previous behaviour.

[behnam-zakeri]

This is the error I get:

Traceback (most recent call last):

  File "<ipython-input-9-6197dafd600e>", line 1, in <module>
    msgSC.add_par('input', df)

  File "C:\Users\zakeri\AppData\Roaming\Python\Python37\site-packages\ixmp\core.py", line 1240, in add_par
    axis=1)

  File "C:\Users\zakeri\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py", line 3037, in __setitem__
    self._set_item(key, value)

  File "C:\Users\zakeri\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\frame.py", line 3114, in _set_item
    NDFrame._set_item(self, key, value)

  File "C:\Users\zakeri\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py", line 3568, in _set_item
    self._mgr.insert(len(self._info_axis), key, value)

  File "C:\Users\zakeri\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\managers.py", line 1159, in insert
    block = make_block(values=value, ndim=self.ndim, placement=slice(loc, loc + 1))

  File "C:\Users\zakeri\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\blocks.py", line 2717, in make_block
    return klass(values, ndim=ndim, placement=placement)

  File "C:\Users\zakeri\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\internals\blocks.py", line 131, in __init__
    f"Wrong number of items passed {len(self.values)}, "

ValueError: Wrong number of items passed 0, placement implies 1

[khaeru]

Can you please report ixmp show-versions, as suggested by the issue template? This may be due to a change in pandas, so it will help to know the version.

[khaeru]

I note that we don't have a test for this behaviour. This test could be parametrized to cover it: https://github.com/iiasa/ixmp/blob/a236ea6d77abb60b8acd3b5f54e09480e0016ea7/ixmp/tests/core/test_scenario.py#L154-L159

[behnam-zakeri]

Thanks @khaeru for the quick response. Here is the output of ixmp show-versions:

$ ixmp show-versions

ixmp:        3.0.0
message_ix:  3.0.2.dev83+g061d0654
     289d802 (HEAD -> master) reporting updated for act, cap and cap_new
message_data: installed
     76fbe7b3 (HEAD -> covid) path for output Excel files added

click:       7.0
dask:        2.23.0
graphviz:    0.13.2
jpype:       0.7.5
pandas:      1.1.0
pint:        0.11
xarray:      0.14.1
xlrd:        1.2.0
xlsxwriter:  1.2.7
yaml:        5.2

iam_units:   installed
jupyter:     installed
matplotlib:  3.2.0
plotnine:    0.7.1
pyam:        0.7.0

python:      3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS:          Windows
OS-release:  10
machine:     AMD64
processor:   Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder:   little
LC_ALL:      None
LANG:        en_US.UTF-8
LOCALE:      None.None

[khaeru]

Here is the output of ixmp show-versions:

Can you include the full output, including the versions or commits of ixmp and message_ix?

[khaeru]

In particular, your traceback includes:

 File "C:\Users\zakeri\AppData\Roaming\Python\Python37\site-packages\ixmp\core.py", line 1240, in add_par
    axis=1)

but current master has that line (axis=1) further down, at line 1296 instead of 1240: https://github.com/iiasa/ixmp/blob/a236ea6d77abb60b8acd3b5f54e09480e0016ea7/ixmp/core.py#L1290-L1298

I opened #374 to add one test with a DataFrame with zero rows, which passes.

Do the example data frames from your code have columns, or are they basically the same as df = pd.DataFrame()?

[behnam-zakeri]

Can you include the full output, including the versions or commits of ixmp and message_ix?

I updated the output of ixmp show-versions, above.

[behnam-zakeri]

Do the example data frames from your code have columns, or are they basically the same as df = pd.DataFrame()?

I get the error even for empty dataframes that have the same columns of that parameter. For example, for parameter input, the empty dataframe looks like:

Empty DataFrame
Columns: [node_loc, technology, year_vtg, year_act, mode, node_origin, commodity, level, time, time_origin, value, unit]
Index: []

[khaeru]

I get the error even for empty dataframes that have the same columns of that parameter.

This seems to be contradicted by the test added in #374. The difference might have to do with the index dimensions/names distinction; I can expand the tests there to check this.