Nandi corrected branch

[NMoksnes]

An initial merge to master. There are still some functions that will be added, but the structure is the same.

[willu47]

Another bug:

% python run_datafile.py data osemosys_shell_param.txt willtest.txt 
Operational life 2020-10-09 16:10:45
Fixed cost 2020-10-09 16:10:45
TotalTechnologyAnnualActivityUpperLimit 2020-10-09 16:10:45
SpecifiedAnnualDemand 2020-10-09 16:10:45
Capital cost dynamic cost 2020-10-09 16:10:45
Capital cost 2020-10-09 16:10:45
Capacity to activity 2020-10-09 16:10:45
Emission activity 2020-10-09 16:10:45
Variable cost 2020-10-09 16:10:45
Input activity 2020-10-09 16:10:45
Outputactivity 2020-10-09 16:10:45
write to file 2020-10-09 16:10:45

% glpsol -m ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt -d willtest.txt --check
GLPSOL: GLPK LP/MIP Solver, v4.65
Parameter(s) specified in the command line:
 -m ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt -d willtest.txt
 --check
Reading model section from ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt...
1112 lines were read
Reading data section from willtest.txt...
willtest.txt:4817: number, symbol, or := missing where expected
Context: ...esidualCapacity default 0 := ; param YearSplit default 0 : ;
MathProg model processing error

[NMoksnes]

@willu47 Great catch with the emissionactivity. It was the input file that I hadn't adjusted to the code.

For the data files they are important that the user defines them the same way (naming and also columns etc) so maybe that would be good to keep. The data is just sample data.

For the unwanted files I will see how to do that in GitHub Desktop...

[NMoksnes]

Another bug:

% python run_datafile.py data osemosys_shell_param.txt willtest.txt 
Operational life 2020-10-09 16:10:45
Fixed cost 2020-10-09 16:10:45
TotalTechnologyAnnualActivityUpperLimit 2020-10-09 16:10:45
SpecifiedAnnualDemand 2020-10-09 16:10:45
Capital cost dynamic cost 2020-10-09 16:10:45
Capital cost 2020-10-09 16:10:45
Capacity to activity 2020-10-09 16:10:45
Emission activity 2020-10-09 16:10:45
Variable cost 2020-10-09 16:10:45
Input activity 2020-10-09 16:10:45
Outputactivity 2020-10-09 16:10:45
write to file 2020-10-09 16:10:45

% glpsol -m ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt -d willtest.txt --check
GLPSOL: GLPK LP/MIP Solver, v4.65
Parameter(s) specified in the command line:
 -m ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt -d willtest.txt
 --check
Reading model section from ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt...
1112 lines were read
Reading data section from willtest.txt...
willtest.txt:4817: number, symbol, or := missing where expected
Context: ...esidualCapacity default 0 := ; param YearSplit default 0 : ;
MathProg model processing error

The file is not ready to run in OSeMOSYS. You need to add all SETs and yearsplit etc. This is not added through the script. Do you think that I should make a "mock up" OSeMOSYS file with the data that is not location specific?

[willu47]

I think there could be an alternative way to help users with data formats. Agree that providing a sample data set is very useful, but perhaps you should aim to document this is a way that allows others to create their own dataset.

Alternatively, using something like frictionless could help automate the process of documenting the data structure... Something to add to the issue tracker.

Another observation is that there is a data folder in tests which seems to be updated by the tests. Is this output data (results) that you are tracking? If so, it probably should not be - and you should use temporary folders within the tests. Again, not a deal breaker and something to work on in the future.

[NMoksnes]

I think there could be an alternative way to help users with data formats. Agree that providing a sample data set is very useful, but perhaps you should aim to document this is a way that allows others to create their own dataset.

Alternatively, using something like frictionless could help automate the process of documenting the data structure... Something to add to the issue tracker.

Another observation is that there is a data folder in tests which seems to be updated by the tests. Is this output data (results) that you are tracking? If so, it probably should not be - and you should use temporary folders within the tests. Again, not a deal breaker and something to work on in the future.

Ok. That sounds good. It is important to be very clear with the formats. But if you think there is an alternate way then let's look into that later on. For energy modellers I think CSV is a very nice and familiar format. Just to use Python and maybe adjusting some code can also be challenging for some modellers.

I'm not sure about the data folder which is updated by the test. Can you point me to the line where that is happening. It should just check the string (outPutFile) that the wanted selected string is there in the right format.

[willu47]

Okay, so the tests/data and data/ seem to contain the same data? Can the data folder be removed, and we keep the tests/data folder? When packaging the programme, neither will be included (users will need to provide their own dataset). And keeping two copies in the repo isn't necessary.

For now, users can look at the data structure in the tests/data folder if they want to see how to structure their data. Eventually, this can be documented better.

Otherwise, I think this is ready to merge now.