Error with preprocess_data.py script

[willu47]

When using the scripts/preprocess_data.py script, I get the following error, using a text file generated from otoole:

$ python scripts/preprocess_data.py otoole output_data/ref5fast.txt output_data/ref5fast_modex.txt 
Traceback (most recent call last):
  File "/Users/wusher/repository/jrc_temba/scripts/preprocess_data.py", line 311, in <module>
    main(data_format, data_infile, data_outfile)
  File "/Users/wusher/repository/jrc_temba/scripts/preprocess_data.py", line 210, in main
    data_all.append(tuple([tech, mode]))
UnboundLocalError: local variable 'tech' referenced before assignment

[willu47]

Also, is it necessary to use this script with the osemosys_fast code? There appear to already be entries in the fast code for MODEx subsets.

[tniet]

Hey @willu47 - the fast code and the pre-processing script are incompatible. The fast code creates the MODEx subsets using MathProg while the script creates them using Python. The Python script is more efficient. Maybe we need to consider the purpose of the fast code? If it's to run models in the most efficient way possible it could be shifted to be script compatible? That would also solve the OSeMOSYS Global issue (https://github.com/OSeMOSYS/osemosys_global/issues/53) of having to use a modified version rather than a release version. @abhishek0208 what do you think?

[willu47]

It's unhelpful to ship the otherwise very useful MODEX script in a way that it can only be used with an undocumented variant of the OSeMOSYS code. I'm in favour of ensuring that the script is compatible with at least the fast, but ideally short and long versions too.

The error mentioned above though appears to be a bug in the script.

[tniet]

Not sure about the error. I'll leave that to @abhishek0208 to take a look at.

Shall I open a new issue on the MODEX script and making things compatible with the fast code?

[abhishek0208]

Thanks @willu47 and @tniet. Fixed this bug.