JeffersonLab / hps-mc

HPS MC toolkit
1 stars 6 forks source link

Add component level logging and other changes #354

Closed JeremyMcCormick closed 1 year ago

JeremyMcCormick commented 1 year ago

This PR adds component level logging and tweaks how logging and configuration works, in general.

Logging can be configured using the following settings in the .hpsmc file:

[HPSMC]
loglevel = INFO
logfile = /tmp/my.log

The loglevel sets the log level and must be valid according to python logging module conventions.

The logfile is an optional parameter for sending all logging output to a file. (Note that logfile can only be used for the "global" rather than component level configuration.)

Individual components can have their log level set using:

[DummyComponent]
loglevel = DEBUG

...where "DummyComponent" should be replaced by the actual class name of the component.

A bunch of confusing configuration and setup having to do with logging was removed from the Job class.

The .hpsmc configuration is now loaded globally and available as hpsmc.global_config. Additional configurations may be supplied using the "-c" argument to the job command line interface.

In testing, I found that the Job class could not be used without calling the function to parse command line arguments, so I updated it so that it could be created manually without the need for argument parsing. This mainly involved adding default arguments where there were none before.

There are a number of other small bug fixes and cleanup included in the PR as well including removal of some dead code.

I tested the changes to logging and config with examples/beam_slic and everything looks fine.

JeremyMcCormick commented 1 year ago

Here is sample logging output using the new scheme:

hpsmc:INFO Config files found: ['/work/hps/hps-mc/scratch/.hpsmc']
hpsmc.job:INFO Reading additional config from: ['/work/hps/hps-mc/scratch/dummy.cfg']
hpsmc.job:INFO Job ID: None
hpsmc.job:INFO Description: HPS MC Job
hpsmc.job:INFO Job configuration:
[HPSMC]
loglevel=INFO

[Job]
dry_run=False
delete_existing=True
enable_file_chaining=True
check_output_files=True
delete_rundir=False

[DummyComponent]
loglevel=DEBUG

hpsmc.job:INFO Loading job script: ../python/jobs/dummy_job.py
hpsmc.job:INFO Job components loaded: ['dummy']
hpsmc.job:INFO No job parameters were specified!
hpsmc.job:INFO Changing to run dir: /work/hps/hps-mc/scratch
hpsmc.job:INFO Executing 'dummy' with command: dummy
================ Component: dummy ================
hpsmc.component.DummyComponent:DEBUG dummy debug
hpsmc.component.DummyComponent:INFO dummy info
hpsmc.component.DummyComponent:WARNING dummy warn
hpsmc.component.DummyComponent:CRITICAL dummy critical
hpsmc.component.DummyComponent:ERROR dummy error
hpsmc.job:INFO Execution of dummy took 0.0058 second(s) with return code: None
hpsmc.job:INFO Job execution took 0.010637 seconds
hpsmc.job:INFO Running cleanup for component: dummy
hpsmc.job:INFO Successfully finished running job: Dummy

The above has global log level (HPSMC) set to INFO with the component at DEBUG.

JeremyMcCormick commented 1 year ago

@tomeichlersmith

Thank you for the detailed suggestions.

I committed all your proposed changes, as they looked good. This should be ready for final review now.