search

ML4GW / aframev2

Detecting binary black hole mergers in LIGO with neural networks
MIT License
5 stars 16 forks source link

Improve logging/output of pipeline #260

Open wbenoit26 opened 1 month ago

wbenoit26 commented 1 month ago

The logging that comes out of law/luigi is confusing for new users. It's quite verbose, and when something goes wrong, it can be hard to tell what happened (or, if it happened in a condor job, where to look to see what went wrong). What can we do to improve the way logging is done?

EthanMarx commented 1 month ago

This might be out of our control. Theres honestly just a learning curve to determining what is a law specific error and what is a task error. law attempts to help you with this with certain messages. If you have any ideas though happy to hear them

wbenoit26 commented 1 month ago

If it's not possible to turn down the number of messages from law, then one thing I was thinking of was trying to be more explicit about what's going on using our own logging. I've noticed myself slipping on the amount of logging statements I have, just because I've already got a good sense of what's going on. Obviously there's a limit to how much we want to output, but I think we could add more messages indicating what's happening at the current moment (and make it clear that the message is coming from Aframe, rather than from law)

EthanMarx commented 1 month ago

Yeah very true - need to figure out how to get a hold of the logger for logging to console from tasks.

For condor jobs theres not much we can do - users can inspect the condor job logs themselves. We can add logging to the depoyment task however