introduce scenario specific names for .log and .lst GAMS files

[volker-krey]

At present, .log and .lst files of MESSAGEix model runs overwrite each other, since the files are generically called MESSAGE_run.log and MESSAGE-MACRO_run.log when running MESSAGE and MESSAGE-MACRO, respectively (and similarly for the .lst files). It is often useful to keep these files after the scenarios have been completed, e.g., to compare with scenarios run as part of the same batch.

I would therefore suggest to introduce scenario specific names to the .log and .lst files, using the same naming convention as for the GDX input and output files. This can either be achieved by renaming the files after completing the scenarios or using GAMS native command line options -lf for .log files and -o for .lst files that allows naming those files. For example, the following code snippet would something along those lines: gams MESSAGE_run.gms -o=Msg_model_scenario.lst -lf=Msg_model_scenario.log (see GAMS documentation).

With many files potentially accumulating, it would probably make sense to move them to a separate folder.

[volker-krey]

As an additional consideration, storing the .log, .lst and possibly the GDX data and output files of selected scenarios (e.g., published scenarios) in the database for improved documentation and reproducibility could be worthwhile. This would have the additional benefit of providing access to these files to all users as they are usually stored locally.

[SiddharthJoshi-Git]

From the Messageix meeting today, there is a module(s) to automate logging in python. e.g. https://pypi.org/project/compressedlogger/ .

[khaeru]

using GAMS native command line options -lf for .log files and -o for .lst files that allows naming those files

The code already contains mechanisms to generate and pass specific command-line options to GAMS; this is how the input and output GDX files are identified to the GAMS code:

https://github.com/iiasa/message_ix/blob/6dee23a383dc035892d6c51af98d78391302456e/message_ix/models.py#L315-L322

I imagine this could easily be extended to set those other options on a per-solve basis.

There is a module(s) to automate logging in python. e.g. https://pypi.org/project/compressedlogger/

For the present issue, the log files in question are generated entirely by GAMS processes that are not accessible to Python, stdlib logging, or anything built on that. It is a good idea to compress the generated GAMS logs/listings after they are created. For this, we can use the standard library directly.

As an additional consideration, storing the .log, .lst and possibly the GDX data and output files of selected scenarios (e.g., published scenarios) in the database for improved documentation and reproducibility could be worthwhile. This would have the additional benefit of providing access to these files to all users as they are usually stored locally.

I don't know if the current Java ixmp_source (probably not) or the new ixmp4 backend (possibly?) support such storage. I would suggest (a) this can be a separate issue and (b) we first document and promote the practice used in the NAVIGATE project, i.e. using symlinks, network drives, the filesystem and/or configurable paths to point the "GAMS log directory" or "GAMS listing directory" to a shared location.

[volker-krey]

I concur that this can be separated into two issues with the log/lst file storage being much easier to deal with. The idea of storing files as part of the scenario object is probably indeed best addressed in the migration to ixmp4 backend. I will raise it in this context.