search

ucgmsim / slurm_gm_workflow

Porting the GM workflow to run on new NeSI HPC (Maintainer: Jonney)
MIT License
0 stars 2 forks source link

Enforced missing log msgs to stdout #490

Closed sungeunbae closed 1 year ago

sungeunbae commented 1 year ago

When add_to_mgmt_queue.py is called from Slurm/PBS script, the logger add_to_queue() tied to masterlog*.txt file in the cybershake root directory is not accessible. So a status change triggered by the script is never logged.

The idea was to print these messages to stdout. Then the .out file from the scheduler job should have these log messages.

Here is a question: by default, a basic logger (= qclogging.get_basic_logger() ) is used, but no idea where the output goes to. I tried to use get_logger() with stdout_printer switch option, but to no avail.

Instead, I devised this quick and dirty solution. Any suggestions welcome.

At least, I get thse messages and I know the update files are indeed written to where it is supposed to.

(python3_nurion) x2568a07@login04: /scratch/x2568a02/UC/RunFolder/Cybershake/v23p8/Runs/Torlesse/Torlesse_REL23> cat hf.Torlesse_REL*_20230807_103837_13335963.out
2023-08-07_13:21:49
Adding task to the queue. Realisation: Torlesse_REL23, process type: 4, status: 3, job_id: 13335963, error: None
Writing update file to /scratch/x2568a02/UC/RunFolder/Cybershake/v23p8/mgmt_db_queue/20230807132204_911213.Torlesse_REL23.4
Successfully wrote task update file

2023-08-07_13:22:16
Adding task to the queue. Realisation: Torlesse_REL23, process type: 4, status: 5, job_id: 13335963, error: None
Writing update file to /scratch/x2568a02/UC/RunFolder/Cybershake/v23p8/mgmt_db_queue/20230807132320_403039.Torlesse_REL23.4
Successfully wrote task update file
sungeunbae commented 1 year ago

Honestly not sure why the get_basic_logger isnt working. I think from memory which is a little shaky with the logging but it should be placed in the main logger when you run execute run_cybershake.

Would be better to see if we can fix the get_basic_logger and logging if it is an issue as it is used alot throughout the workflow and the message stuff I find a bit messy

Yes, agreed. hope I can get to the bottom of it, as you pointed out, the logging is a little messy.

As ithe logging issue happens only when add_to_mgmt_queue.py is executed as a standalone script (from slurm or PBS ), I think it is probably best to pipe the logging to stdout, so that it will be added to .out, not as a separate log file or not to one of the log files in the root (finding the correct log file name at the stage is not super-straightforward when this script is called from slurm/PBS script). I tried this avenue, but not successful and ended up with this quick and dirty solution.

Your feedback reinforced my initial inclination to fix this properly. I'm happy to withdraw this PR and try a better solution.

sungeunbae commented 1 year ago

Lets' make this PR pending