search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
97 stars 49 forks source link

logs both in Grafana and through Jenkins config - breaks displayed logs (colors?) #918

Closed ArieLevs closed 8 hours ago

ArieLevs commented 1 month ago

Jenkins and plugins versions report

Environment ```text plugin version 3.1320.v2eededb_d909e ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins 2.452.2 Jenkins in kubernetes, default agent is jenkins/inbound-agent:jdk17

Reproduction steps

  1. follow Storing Jenkins Pipeline Logs in Loki
  2. configure the Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins option
  3. logs are shown in Jenkins UI and in Grafana, which is OK
  4. logs are missing colors, links are not clickable (from Jenkins UI), seems that anything that has special characters no longer appears in logs

Expected Results

logs should keep appearing as usual both in Grafana and Jenkins UI

Actual Results

logs are missing colors, links are not clickable (from Jenkins UI), seems that anything that has special characters no longer appears in logs

for example, printed error from tools like tflint, flake8, etc no longer appear to the user

Anything else?

No response

Are you interested in contributing a fix?

No response

cyrille-leclerc commented 1 month ago

We probably know the limitation but I would like to double check please: can you provide a description of the color highlighting and hyperlinks that you are looking for, ideally with an annotated screenshot of pipeline logs with the formatting you are looking for.

The cause is likely to be that Elasticsearch, the first OTel logs backend we supported, didn't support ANSI color invisible chars so we had to extract the rich text formatting annotations from the plain text.

See :

ArieLevs commented 1 month ago

Hi @cyrille-leclerc sure,

here is a screenshot using regular console logs with no otel integration Screenshot 2024-08-13 at 9 33 00

then, when enabling Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins logs look like:

cyrille-leclerc commented 1 week ago

Update: dealing with logs formatting in logs backends that support ANSI color encoding is more complex than anticipated because Jenkins log formatting is no ANSI color encoding or is more than just ANSI color encoding. I'll continue to explore this.

@ArieLevs can you please open a bug about the issue

Caught exception evaluating: it.writeLogTo(offset,output) in /job/xxxx/job/xxxx/view/change-requests/job/PR-148/4/console. Reason: com.jayway.jsonpath.PathNotFoundException: Expected to find an object with property ['data'] in path $ but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

It seems to be specific to your Loki version. Do you use self managed Loki? Which version please?

kuisathaverat commented 6 days ago

It was a designed decision to store logs without the format flags; if you want to keep that format in the Jenkins console view, you can use the mirror option. In this way you send the logs to Loky and keep a local copy in the original way so the format is not lost.

ArieLevs commented 8 hours ago

Update: dealing with logs formatting in logs backends that support ANSI color encoding is more complex than anticipated because Jenkins log formatting is no ANSI color encoding or is more than just ANSI color encoding. I'll continue to explore this.

@ArieLevs can you please open a bug about the issue

Caught exception evaluating: it.writeLogTo(offset,output) in /job/xxxx/job/xxxx/view/change-requests/job/PR-148/4/console. Reason: com.jayway.jsonpath.PathNotFoundException: Expected to find an object with property ['data'] in path $ but found 'java.lang.String'. This is not a json object according to the JsonProvider: 'com.jayway.jsonpath.spi.json.JsonSmartJsonProvider'.

It seems to be specific to your Loki version. Do you use self managed Loki? Which version please?

Done https://github.com/jenkinsci/opentelemetry-plugin/issues/946 Yes using self managed Loki 3.0.0 (k8s)

ArieLevs commented 8 hours ago

It was a designed decision to store logs without the format flags; if you want to keep that format in the Jenkins console view, you can use the mirror option. In this way you send the logs to Loky and keep a local copy in the original way so the format is not lost.

this actually worked very well, the only thing is I can no long see a "link" to Grafana, similar to how its used with Store pipeline logs In Loki and visualize logs both in Grafana and through Jenkins.
to get the desired logs, I had to randomly search a string that appeared in my console logs, then get the trace_id and only then filter by that.

I'll mark issue as closed as all is actually working as expected, it just feels like end uses of Jenkins will never use this feature, as they don't have a direct link from Jenkins <-> Grafana this way. (but this is still good enough for our infra team to investigate Jenkins and get insights)