search

lsst-sims / legacy_sims_movingObjects

3 stars 5 forks source link

non-unique log filenames #11

Closed rhiannonlynne closed 7 years ago

rhiannonlynne commented 7 years ago

The log file names are just the original orbit file + __log, so if you run over multiple timespans or different objects, the old log files are erased and overwritten. Or worse (?), if you run the same orbit file but with different parameters (objects or timespans) at the same time, both processes will try to write to the same output log file.

I'm not sure how useful the log file is, so I'm not sure if this is something that is actually problematic. It has been for me, but I may be running a pretty atypical use case and there may be reasons to keep the single log file name. @danielsf ?

danielsf commented 7 years ago

Given that I generated all of the MBA coefficients before we had this repo, I don't have any experience to inform an opinion. Can this be fixed by specifying an output directory to which all of the output files and logs go? You could then just make sure that each unique run has its own output directory (sorry if that is a solution you have already tried).

rhiannonlynne commented 7 years ago

No, I didn't try that .. I didn't have any reason to separate the chunks. (basically, I ran a bunch of timesteps at once).

However, I saw that this is something you'd like as a separate issue. So, I propose that generateCoefficients should generate log filenames that match the whole coeff and resid file names (with timesteps and object ids if applicable), and that we add output directory support. I just wanted to check there wasn't some reason you wanted the log files to have the same names.

danielsf commented 7 years ago

The log files in the previous version of the code weren't as helpful as the slurm log files produced by NERSC, so I did not really look at them. Maybe just include the start mjd and end mjd in the log file name, so if you do a bunch of timesteps in parallel, they don't overwrite each others' log files. People running different sets of parameters at the same time will just have to be careful.

rhiannonlynne commented 7 years ago

Updated log file names to include some portion of time.