Guide for configuring for Python log files

[jlcash]

Guide for configuring for Python log files

I thought the community would benefit from a quick walkthrough on how to configure for Python. After spending a few hours getting this to work, I have documented the steps required to build up the log format from the format configured for python logging.

My python logging format is configured in a yaml file and is as follows:

precice:
        format: "%(asctime)s - %(threadName)s - %(name)s - %(levelname)s - %(message)s"
        datefmt: '%Y-%m-%d %H:%M:%S'

Starting with the easiest first, in the ideolog settings I configured:

• The message start pattern as ^\d{4} to match the start of the line followed by the 4 numbers that make up the year of the timestamp (yyyy); and
• The time format as yyyy-MM-dd HH:mm:ss to match the datefmt above.

I then moved on to the message pattern. You will see that there are 5 distinct groups from the format specified above, each separated by space dash space. I built up my log format as follows:

Start of line ^

Followed by first group that contains the timestamp in the format specified by datefmt above. For this I used 4 numbers dash 2 numbers dash 2 numbers space 2 numbers colon 2 numbers colon 2 numbers. These are all contained in brackets as they will be referred to later as the time capture group: (\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})

I then added the space dash space separator outside of the capture group: \s-\s

Then for the next capture group which in my case contains the thread name. This is simple as it is just an unknown length string: (.*)

Then another separator followed by another unknown length string capture group for the class name and another separator: \s-\s(.*)\s-\s

Then the capture group containing the log level. In my case this is between 4 and 7 uppercase letters (INFO, ERROR, DEBUG or WARNING): ([A-Z]{4,7})

Then another separator followed by a unknown length string for the log message: \s-\s(.*)

Followed by end of line: $

Putting this all together gives us the following message pattern: ^(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s-\s(.*)\s-\s(.*)\s-\s([A-Z]{4,7})\s-\s(.*)$

I then set up the capture groups. Time=1, Severity=4, Category=3

Finally I set up the patterns for log level. ^\sERROR\s$ as highlight field red + stripe ^\sWARNING\s$ as highlight field orange ^\sINFO\s$ as highlight field green.

After closing and reopening my log file everything worked as expected.

[ZeroCool2u]

Okay, this message pattern works as is, but the following new patterns are required for the log level highlighting to work properly. Also, if you want to use the CRITICAL messages built into the Python logger, you'll need to adjust the message pattern very slightly, to allow for CRITICAL to be recognized, because it is 8 characters instead of 7. The message pattern below will work for that. Note that the only difference is in the final capture group, ([A-Z]{4-7}) -> ([A-Z]{4-8}).

^(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s-\s(.*)\s-\s(.*)\s-\s([A-Z]{4,8})\s-\s(.*)$

^\s*C(RITICAL)?\s*$
^\s*E(RROR)?\s*$
^\s*W(ARN(ING)?)?\s*$
^\s*I(NFO)?\s*$

Here is a basic logging setup that you can drop in at the beginning of your program and will work with this message pattern.

import logging
logging.basicConfig(filename=r'logs/app_logs.log', format='%(asctime)s - %(threadName)s - %(name)s - %(levelname)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S', level=logging.INFO)

[joveice]

I don't get this, I'm using the logging example from @ZeroCool2u with the regex but I get "Log format not recognized" after i reopen the file and after a restart.

logging.basicConfig(filename='app.log', filemode='a', level=logging.INFO,
                    format='%(asctime)s - %(threadName)s - %(name)s - %(levelname)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

Log:

2019-12-15 20:17:02 - MainThread - root - INFO - Startup.
2019-12-15 20:17:02 - MainThread - root - INFO - Finished.

Message pattern ^(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})\s-\s(.*)\s-\s(.*)\s-\s([A-Z]{4,8})\s-\s(.*)$ Message start pattern ^\d{4} Time format yyyy-MM-dd HH:mm:ss Time cap: 1, Severity: 4, Category: 3

How does this work?

[ZeroCool2u]

There are inconsistencies with the line endings. Try hitting enter at the end of the log file or deleting the last line of the file.

[joveice]

Thanks @ZeroCool2u that indeed made it parse it.

[ZeroCool2u]

@joveice No worries! Seems to work consistently for me most of the time if I don't touch the log file, especially with ones larger than the max file size limit set for your IDE.

[pjm4github]

For those that need a screenshot here is one.

[NTierSoftware]

I agree, plugin is too difficult to use to be of any value.

[D4rk-S0ul]

My logging format: format="%(asctime)s - %(levelname)s: %(message)s", datefmt="%d.%m.%Y: %I:%M:%S %p"

What I entered in log formats: Message Pattern: ^(\d{2}.\d{2}.\d{4}:\s\d{2}:\d{2}:\d{2}\s[A-Z]{2})\s-\s([A-Z]{4,8}):\s(.*)$ Message Start Pattern: ^\d{2} Time Format: dd.MM.yyyy: HH:mm:ss a

All a that seems fine to me, but for whatever reason nothings working (Log format not recognized). Any ideas?

[mfilippov]

@D4rk-S0ul Could you attach a sample of your log?