oemof.tabular CSV files: "output_parameters" cannot be formatted as expected

[unndreay]

When introducing more than one output_parameter into one of the CSV files or a parameter with a dict where a comma is needed:

region,name,type,carrier,tech,from_bus,to_bus,capacity,capacity_cost,efficiency,carrier_cost,marginal_cost,expandable,output_parameters
AT,AT-ch4-gt,conversion,ch4,gt,AT-ch4,AT-electricity,0,52696.25833435559,0.465,70.92530000000001,1.1,True,{"negative_gradient": {"ub": 1000, "costs": 0}}

(see {"negative_gradient": {"ub": 1000, "costs": 0}} at the end), an error is raised:

19:15:48-INFO-Used oemof version: 0.3.2
19:15:48-INFO-Start running the whole modelling pipeline
19:15:48-INFO-Creating EnergySystem from datapackage
Traceback (most recent call last):
  File "/home/unndreay/Workspaces/oemof-tabular/src/oemof/tabular/datapackage/reading.py", line 135, in deserialize_energy_system
    r.read()
  File "/home/unndreay/.virtualenvs/oemo-flex/lib/python3.7/site-packages/datapackage/resource.py", line 188, in read
    return self.__get_table().read(relations=relations, **options)
  File "/home/unndreay/.virtualenvs/oemo-flex/lib/python3.7/site-packages/tableschema/table.py", line 133, in read
    for count, row in enumerate(rows, start=1):
  File "/home/unndreay/.virtualenvs/oemo-flex/lib/python3.7/site-packages/tableschema/table.py", line 80, in iter
    for row_number, headers, row in iterator:
  File "/home/unndreay/.virtualenvs/oemo-flex/lib/python3.7/site-packages/tableschema/table.py", line 191, in builtin_processor
    row = self.__schema.cast_row(row)
  File "/home/unndreay/.virtualenvs/oemo-flex/lib/python3.7/site-packages/tableschema/schema.py", line 138, in cast_row
    raise exceptions.CastError(message)
tableschema.exceptions.CastError: Row length 15 doesn't match fields count 14

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/unndreay/Workspaces/oemo-flex/experiment_1/scripts/FlexMex1_2a/runall.py", line 16, in <module>
    optimization.main()
  File "/home/unndreay/Workspaces/oemo-flex/experiment_1/scripts/FlexMex1_2a/optimization.py", line 37, in main
    attributemap={}, typemap=TYPEMAP,
  File "/home/unndreay/Workspaces/oemof-tabular/src/oemof/tabular/datapackage/reading.py", line 137, in deserialize_energy_system
    raise dp.exceptions.CastError((cast_error_msg).format(r.name))
tableschema.exceptions.CastError: Metadata structure of resource `ch4-gt` does not match data structure. Check the column names, types and their order.

oemof.tabular's reading function interprets the comma as a field separator.

We should consider using semi-colons as field separators as in the oemof.tabular examples. The following works.

region;name;type;carrier;tech;from_bus;to_bus;capacity;capacity_cost;efficiency;carrier_cost;marginal_cost;expandable;output_parameters
AT;AT-ch4-gt;conversion;ch4;gt;AT-ch4;AT-electricity;10000;52696.25833435559;0.465;70.92530000000001;1.1;False;{"negative_gradient": {"ub": 1000, "costs": 0},"positive_gradient": {"ub": 1000, "costs": 0}}

[jnnr]

What is the status here? Is this obsolete as we do not use output_parameters any more as they do not go along with InvestmentFlows? Should we switch to semicolons anyway?

[jnnr]

Does not seem to apply to the cases we calculate. We can make semicola a convention for input data. A short sentence in the docs would be good. I move this to the next milestone.

[unndreay]

Input parameters are used in h2-cavern and electricity-bev now. There is only one field, no comma needed, see https://github.com/modex-flexmex/oemo-flex/pull/81#issuecomment-697371252 But I agree to doc it.

[jnnr]

This is a TODO in the docs issue now, so I close this one.