search

OSeMOSYS / otoole

OSeMOSYS Tools for Energy
https://otoole.readthedocs.io
MIT License
25 stars 19 forks source link

Converting atlantis data file into csv is crashing #21

Closed neunzehnachtneun closed 5 years ago

neunzehnachtneun commented 5 years ago

Hi together, I am using otoole v0.4.1 and as Pr #7 is fully merged, I expected converting of the atlantis data file into csv should work. Unfortunately I get the following error messages. pyparsing v2.4.2 pulp v1.6.8

Do you have any suggestions or ideas?

Kind, Sebastian

otoole convert datafile datapackage atlantis_bau.txt atlantis
Traceback (most recent call last):
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/bin/otoole", line 10, in <module>
    sys.exit(main())
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/otoole/cli.py", line 165, in main
    args.func(args)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/otoole/cli.py", line 68, in conversion_matrix
    convert_file_to_package(args.from_path, args.to_path)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/otoole/preprocess/datafile_to_datapackage.py", line 30, in convert_file_to_package
    amply_datafile = read_in_datafile(path_to_datafile, config)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/otoole/preprocess/datafile_to_datapackage.py", line 55, in read_in_datafile
    datafile_parser.load_file(datafile)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pulp/amply.py", line 772, in load_file
    self.load_string(f.read())
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pulp/amply.py", line 763, in load_string
    for obj in grammar.parseString(string):
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 1939, in parseString
    raise exc
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 1929, in parseString
    loc, tokens = self._parse(instring, 0)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 1830, in _parseCache
    value = self._parseNoCache(instring, loc, doActions, callPreParse)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 1669, in _parseNoCache
    loc, tokens = self.parseImpl(instring, preloc, doActions)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 4037, in parseImpl
    loc, exprtokens = e._parse(instring, loc, doActions)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 1830, in _parseCache
    value = self._parseNoCache(instring, loc, doActions, callPreParse)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 1673, in _parseNoCache
    loc, tokens = self.parseImpl(instring, preloc, doActions)
  File "/Users/someuser/miniconda3/envs/esm_py37_dev/lib/python3.7/site-packages/pyparsing.py", line 3783, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected end of text, found '#'  (at char 0), (line:1, col:1)
neunzehnachtneun commented 5 years ago

I just checked, if this error also occurs converting the utopia data file. The error persists.

willu47 commented 5 years ago

Hi @neunzehnachtneun Thanks a lot for the bug report. This problem is due to the amply module in v1.6.8 of the PuLP package not yet ignoring lines in the OSeMOSYS file which begin with a # e.g. commented lines (see https://github.com/coin-or/pulp/issues/222)

One way to avoid this is to install a development version of PuLP. You can do this as follows:

git clone https://github.com/coin-or/pulp
cd pulp
python setup.py develop

Hopefully, the PuLP developers will release a new version of PuLP soon.

willu47 commented 5 years ago

An alternative pip command which means you don't need to clone the code:

pip install -e git://git.coin-or.com/pulp.git@master#egg=pulp

neunzehnachtneun commented 5 years ago

Wow, thanks for the quick reply! For us this is not really a solution, as we need a common and consistent environment on multiple machines. But I tried to delete all lines beginning with a #. It seems, that also the last line end; is also causing a parsing error.

But deleting also this line gets the script running.

willu47 commented 5 years ago

Hi @neunzehnachtneun, my pleasure. Thanks for making use of the otoole library.

Do note that releases of otoole prior to v1.0 are likely to be backward incompatible and see many changes to the API as we develop it further. We'll try our best to make changes thoughtfully to cause minimal disruption but cannot make any promises.

We would really value any feedback or uses you might have for this package as well.

Out of interest, are you using the command line interface, or directly calling the Python modules?

willu47 commented 5 years ago

Changeset coin-or/pulp#230, now merged into the PuLP codebase should fix the above issues with comments # and end statements in the atlantis datafile.