Lossless Logging Localization

[Andesha]

In the old version there was a fairly robust logging system for each stage that was being executed.

Previously I believe this was just information being printed to stdout and then redirected to a file through a SLURM option...

What we should do instead is some of the following:

• Figure out what from the MNE side of things is being written to stdout and find out if we need it for logging
  • We can capture this somehow I think
• Agree on the location of these files
  • I propose the same as before at derivatives/lossless/logs type folder
• Create the name of the file based on $FILENAME_$TIMESTAMP.ll_log
• Prepend the file with the contents of the config being used to run it, just in case
• As a first step, wrap the various function calls inside of _run in pipeline.py with basic logging calls like
  • "Setting Montage", "Finding Breaks", "Flagging comically bad outliers", etc
• Add some timing information in these steps as well for profiling on clusters in the future

[scott-huberty]

Figure out what from the MNE side of things is being written to stdout and find out if we need it for logging

• We can capture this somehow I think

MNE uses a function mne.utils.logger. The verbosity level can be set by the user/program too (standard, debug, etc): https://mne.tools/stable/logging.html

We could just use mne.utils.logger in our pipeline too (i.e. logger.info(), logger.debug(). ) you'll see in pylossless.pipeline.py that I already imported this logger.

But if you want to suggest a different tool (for example logging in the standard library) i'm open to that too.

[scott-huberty]

Bonus points if we can include some emojis like :white_check_mark: for major steps of the pipeline ; )

[Andesha]

Bonus points if we can include some emojis like white_check_mark for major steps of the pipeline ; )

I don't know what that would do on the cluster, but let's try :laughing:

We could just use mne.utils.logger in our pipeline too (i.e. logger.info(), logger.debug(). ) you'll see in pylossless.pipeline.py that I already imported this logger.

I had forgotten (or not noticed?), but that sounds just right!

[christian-oreilly]

Yes, indeed, it should definitely rely and extend on MNE logging utils.... I thought we discussed this in a separate issue, but could not find it. Maybe we just discussed it and never did anything with it.

Note @Andesha that we did not implement anything specifically SLURM execution. It can be used in a SLURM environment more or less easily by calling the python script from BASH, but I had in mind (although not high in my priority list) to include a somewhat more user-friently integration (i.e., adding the SLURM conf into our YAML project conf and launching the SLURM job directly through Python)