search

pegasystems / pega-logviewer

The log viewing tool helps users understand and debug Pega log files. To download the tool, click on the "release" link, scroll to the bottom, and download the logviewer zip file.
Apache License 2.0
33 stars 11 forks source link

Support for JSON Logs #16

Open micgoe opened 8 months ago

micgoe commented 8 months ago

Problem We are currently evaluating a change in our logging infrastructure. We have Open Search in place, which we do all our analysis on. At the moment, we simultaneously store logs in a non-JSON format on a per-pod basis on shared network storage. With our new OpenSearch logging solution in place, we are considering shutting down our old shared network storage solution. The only reason we have not done so is that we are using these logs in none-JSON format for Pega support tickets.

The internal Pega support currently relies on the Pega-logviewer. If the logviewer undergoes enhancement to accept JSON logs alongside the traditional "original" Pega logs, it would empower us and other clients to seamlessly transition to modern logging solutions. Consequently, there would be no need to retain "old" log files solely for the sake of Pega support.

Request Add a feature to the log viewer supporting JSON logs in a JSON format defined as per your "pega-helm-chart/pega" Helm Chart (https://github.com/pegasystems/pega-helm-charts/blob/9ccd6b510493e83d7a33c56cfee2d886491c03b0/charts/pega/config/deploy/prlog4j2.xml#L5)

vargm commented 8 months ago

Hi @micgoe , Thanks for your report of this issue. Logviewer tool can parse different types of Pega log format(Rules,Alert, Cluster, Dataflow, etc). Depending on log type, there are different views/post processing of data. There is already some support built-in for Json formatted logs(specific to logs from Pega Cloud) but the logs are already segregated based on type, different files for Rules, alert, dataflow etc.. The issue here is that the appender you referenced is a "console" appender, and therefore the file is a mix of all types of output. for ex, below lines have both Rules and Alert log entry

{"app":"","stack":"","source_host":"abc2-web-0","level":"INFO","appender_ref":"PEGA","message":"Using codeset version and timestamp map: {pega-enginecode=08-06-07@2023-11-10 09:28:00.0, customer=06-01-01@1970-01-01 00:00:00.0}","userid":"","pegathread":"","@timestamp":"2023-11-20T11:03:59.320Z","thread_name":"main","@version":1,"tenantid":"","logger_name":"com.pega.pegarules.session.internal.mgmt.PRNodeImpl"}
{"app":"","stack":"","source_host":"abc2-web-0","alertType":"performance","level":"ALERT","appender_ref":"PEGA","message":"2023-11-20 11:04:00,655 GMT*8*PEGA0005*3172*500*NA*NA*NA*NA*NA*NA*NA*NA*?*-1**0*NA*NA*com.pega.pegarules.data.internal.store.DataStorePreparedStatement*NA*NA*NA*NA*NA*false*NA**NA*NA*NA*NA*NA*NA*NA*NA*Database query took more than the threshold of 500 ms: 3,172 ms\tSQL: select pzPVStream, pzTenantId from pega_rules.pr4_base where pxObjClass = ?*","userid":"","pegathread":"","@timestamp":"2023-11-20T11:04:00.667Z","thread_name":"StartUp-TaskExecutor","@version":1,"tenantid":"","logger_name":"com.pega.pegarules.data.internal.store.DataStorePreparedStatement"}

So far some of my initial thoughts is on how to parse such a log file, ie, should the tool open the file as a "Rules" log file or a Alert log file (and discard other log type entries). or open multiple tabs for each log type. something i will need to explore around this for now.

Thanks