Make logging more concise

[oavdeev]

Just wanted to open this discussion to solicit ideas how to make it a little more compact:

90 character prefix (or 104 characters when using local metadata) is a lot even on a big screen.

If I had to prioritize these bits of info by importance, I'd say

• step name and task id are absolutely required (without them you can't tell apart different log streams)
• run id is very useful but we could technically print it once. However it is nicer that you have something that you can easily copy and paste when retrieving artifacts and results
• timestamp is convenient but not strictly required, and we could probably remove milliseconds
• batch job id is occasionally useful for debugging if you want to look the task up in AWS console, but not sure if people do this that often (and it takes a lot of screen space)
• worker PID i'm not sure what i'd use for as a user?

Thoughts?

[savingoyal]

One idea could be to introduce logging options on CLI - something like python flow.py --logging=concise run or maybe python flow.py --no-verbose run. What do you think?

[oavdeev]

Yep that's the baseline solution. However I'd like to spend some time to think if there's a way to improve the default behavior here.

Adding more options is fine (you could even imagine making the columns customizable), but it will only affect the experience for the few people that will use them. And I want to be mindful about increasing overall testing surface, in the end it may not be worth added complexity (however small) if only a few users will benefit from this.

[duanckham]

Any progress here?