search

iiasa / Condor_run_R

R scripts to submit and analyse HT Condor runs
BSD 3-Clause "New" or "Revised" License
6 stars 2 forks source link

strong preference #3

Closed d-leclere closed 3 years ago

d-leclere commented 3 years ago

I would leave (gdx={GDX_OUTPUT_DIR}/{GDX_OUTPUT_FILE}) out of the following default arguments to the condor_run.R file: GAMS_ARGUMENTS = "gdx={GDX_OUTPUT_DIR}/{GDX_OUTPUT_FILE} //nsim='%1' PC=2 PS=0 PW=130" # additional GAMS arguments, can use {} expansion here The reason is that getting this gdx back should be done only when we want it (i.e., option, not by default), and actually in that case we have in the 6 some code instructions that generate it (the purpose being to output a gdx file that contains only desirable parameters, not everything, as too heavy to merge). The current setup makes it that the gdx created by this command line argument to the gams execution puts the entire memory dump into a gdx that actually erases a potentially already created gdx (that is much smaller and targeted to what we want). Another fix would be simply to change the name (so that this gdx does not erase the other one generated by the 6), but I would simply remove this command line argument, because if people want the memory dump, they can get the g00 outputs back as well. Hope this is clear.

acjbrouwer commented 3 years ago

The reason is that this repository is intended to be general-purpose instead of GLOBIOM-specific: it is public. For submitting arbitrary GAMS jobs, the GAMS_ARGUMENTS default with gdx=... is more suitable since GDX writing code cannot be assumed to be present, and this default makes it explicit where the GDX is expected to come from.

Of course it would be nice to have a default configuration for GLOBIOM, but that exists, can be found in the GLOBIOM Trunk at R/sample_config.R, and works with the Trunk without further adjustments. As you can see there, the gdx= argument is absent, and a few other GLOBIOM specifics are present. Using that configuration for the Condor_run.R in-script defaults would be a rather peculiar starting point for someone intending to schedule an arbitrary GAMS job.

To prevent others from running into the conflict you ran in to, I instead prefer to document the location of the GLOBIOM sample configuration in the Use section of the README.md and maybe give some additional guidance on output generation and configuration near there.

d-leclere commented 3 years ago

Thanks for the rapid answer. I suppose the GLOBIOM Trunk R/sample_config.R is kept up to date/fully compatible with latest evolution of Condor_runs.R. If yes then I am happy, also noted the link both this rep and to GLOBIOM Trunk's R/sample_config.R is clearly explained in the GLOBIOM wiki.

acjbrouwer commented 3 years ago

It is kept up-to-date both with Condor_run.R-latest and the Trunk itself. Potentially with a bit of delay since not all that happens to the Trunk is known to me as it happens.

acjbrouwer commented 3 years ago

I have added a note on sample_config.R to the README and also added some explanation of the parallel submission constraint.. See commit 3b8d8bc136683d6bd6cbe21dde88863e53041845.