Pipeline logs not shown in jenkins build console

shravanipatil527 commented 1 year ago

Jenkins and plugins versions report

Jenkins: 2.401.1
OS: Linux - 4.18.0-348.2.1.el8_5.x86_64
Java: 11.0.13 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.18.0
Parameterized-Remote-Trigger:3.1.5.1
ace-editor:1.1
ansicolor:1.0.2
ant:481.v7b_09e538fcca
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
authentication-tokens:1.4
bitbucket:223.vd12f2bca5430
bitbucket-build-status-notifier:1.4.2
bitbucket-pullrequest-builder:1.5.0
blueocean:1.26.0
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.5
blueocean-commons:1.27.5
blueocean-config:1.26.0
blueocean-core-js:1.27.5
blueocean-dashboard:1.26.0
blueocean-display-url:2.4.1
blueocean-events:1.26.0
blueocean-git-pipeline:1.26.0
blueocean-github-pipeline:1.26.0
blueocean-i18n:1.26.0
blueocean-jira:1.26.0
blueocean-jwt:1.27.5
blueocean-personalization:1.26.0
blueocean-pipeline-api-impl:1.27.5
blueocean-pipeline-editor:1.26.0
blueocean-pipeline-scm-api:1.27.5
blueocean-rest:1.27.5
blueocean-rest-impl:1.27.5
blueocean-web:1.27.5
bootstrap4-api:4.6.0-5
bootstrap5-api:5.3.0-1
bouncycastle-api:2.27
branch-api:2.1122.v09cb_8ea_8a_724
build-name-setter:2.2.0
build-timeout:1.20
build-with-parameters:73.v3ed13cb_d6cca
built-on-column:1.1
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloudbees-bitbucket-branch-source:784.v7fcdc7c670f6
cloudbees-folder:6.815.v0dd5a_cb_40e0e
command-launcher:90.v669d7ccb_7c31
commons-httpclient3-api:3.1-3
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-36.vc008c8fcda_7b_
conditional-buildstep:1.4.2
copyartifact:1.46.4
credentials:1271.v54b_1c2c6388a_
credentials-binding:631.v861c06d062b_4
delete-log-plugin:1.0
display-url-api:2.3.9
docker-commons:1.21
docker-slaves:1.0.7
docker-workflow:1.28
durable-task:513.vc48a_a_075a_d93
echarts-api:5.4.0-5
email-ext:2.88
envinject:2.4.0
envinject-api:1.8
extended-choice-parameter:346.vd87693c5a_86c
external-monitor-job:203.v683c09d993b_9
favorite:2.3.3.1
file-operations:1.11
font-awesome-api:6.4.0-2
generic-webhook-trigger:1.86.3
git:5.2.0
git-client:4.4.0
git-parameter:0.9.18
git-server:1.11
github:1.37.3
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1732.v3f1889a_c475b_
google-container-registry-auth:0.3
google-kubernetes-engine:0.6.3
google-oauth-plugin:0.8
groovy-label-assignment:1.2.0
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
http_request:1.15
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-8
jaxb:2.3.8-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.27.5
jenkins-multijob-plugin:611.v9d3180d752e6
jira:3.7.1
jjwt-api:0.11.5-77.v646c772fddb_0
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.7.0-1
jsch:0.1.55.61.va_e9ee26616e7
junit:1217.v4297208a_a_b_ce
label-verifier:1.4
ldap:2.9
lockable-resources:2.18
login-theme:1.1
mailer:457.v3f72cb_e015e5
material-theme:0.4.1
matrix-auth:3.1.5
matrix-project:789.v57a_725b_63c79
maven-plugin:3.16
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
momentjs:1.1.1
monitoring:1.94.0
msbuild:1.30
multiple-scms:0.8
nexus-artifact-uploader:2.13
nodelabelparameter:1.10.3.1
oauth-credentials:0.5
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
opentelemetry:2.13.0
pagerduty:0.7.1
pam-auth:1.10
parameterized-trigger:2.44
pipeline-build-step:505.v5f0844d8d126
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.28
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.28
pipeline-utility-steps:2.15.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
powershell:1.8
publish-over:0.22
publish-over-ssh:1.24
pubsub-light:1.16
rebuild:1.34
resource-disposer:0.20
run-condition:1.5
rundeck:3.6.13
saferestart:0.7
scm-api:676.v886669a_199a_a_
script-security:1251.vfe552ed55f8d
sectioned-view:1.25
simple-theme-plugin:103.va_161d09c38c7
skip-certificate-check:1.0
slack:631.v40deea_40323b
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.25
ssh-agent:1.24.1
ssh-credentials:305.v8f4381501156
ssh-slaves:1.821.vd834f8a_c390e
ssh-steps:2.0.0
sshd:3.249.v2dc2ea_416e33
structs:324.va_f5d6774f3a_d
swarm:3.34
theme-manager:193.vcef22f6c5f2b_
throttle-concurrents:2.10
timestamper:1.17
token-macro:384.vf35b_f26814ec
trilead-api:2.84.v72119de229b_7
uno-choice:2.6.1
variant:59.vf075fe829ccb
windows-slaves:1.8.1
workflow-aggregator:2.7
workflow-api:1251.vd4889a_b_0a_065
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3726.v83f8cff396c9
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1278.v94b_dc2b_50c6f
workflow-job:1316.vd2290d3341a_f
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.43

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

Enabled the option to Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins.
Logs are visible in elastic but not in jenkins console output

Expected Results

It should show the build logs in console output

Actual Results

It shows nothing. Gives only a blank screen

Anything else?

Opentelemetry configuration:

shravanipatil527 commented 1 year ago

Hello,

We also checked for other option : Store pipeline logs In Elastic and visualize logs exclusively in Elastic (logs no longer visible through Jenkins screens). Here it should gives us "hyperlink to the logs visualization in Kibana" but this is also not working. It still gives blank screen.

The only option that works for us is: Don't store pipeline logs in Elastic.

cyrille-leclerc commented 1 year ago

What is the result of the "validate elasticsearch configuration" button in the config page? See:

https://github.com/jenkinsci/opentelemetry-plugin/blob/master/docs/build-logs.md#storing-jenkins-pipeline-logs-in-elastic-visualizing-logs-both-in-kibana-and-through-the-jenkins-build-console

shravanipatil527 commented 1 year ago

cyrille-leclerc commented 1 year ago

Can you confirm you can see the logs in Elasticsearch and kibana?
Do you see any error message in Jenkins logs?

shravanipatil527 commented 1 year ago

1) Yes, I can confirm. When "Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins" option is used it only shows logs only in Elasticsearch and kibana but not on Jenkins. 2) No error logs are seen in in Jenkins logs.

npsullia commented 1 year ago

i am having similar issue with jenkins logs not getting pushed through open telemetry. however, i see pipeline stages

kuisathaverat commented 1 year ago

@shravanipatil527 Can your Jenkins host connect to Elasticsearch? In any case, there must be errors in the logs of some kind when you access the console log. JEnkns agents and Jenkins Controller need visibility to Elasticsearch to send and retrieve the logs.

the stack trace in logs will be similar to the one at https://github.com/jenkinsci/opentelemetry-plugin/issues/421

kuisathaverat commented 1 year ago

Confirmed in 2.16, the logs are only displayed in BlueOcean and Elasticsearch, but not in the Console output.

kuisathaverat commented 1 year ago

It is related to the changes to mirror the logs locally at https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.10.0 Now you have to specify the property otel.logs.mirror_to_disk=true in advanced settings. So, the option to store logs locally from the backend Elastic Stack does nothing locally now. The logic was removed.

Enabling the local mirror you lose the link to the logs in the Console output, you still send the logs to Elasticsearch but you have to find them manually.

kuisathaverat commented 1 year ago

So we have to remove the option to store logs locally from the Elastic Stack backend, make clear how to enable local mirror of the logs, and avoid losing the link to the Kibana logs when the Elastic Stack backend is enabled.

Geni2 commented 1 year ago

Hi @kuisathaverat I have been working on the implementation of OpenTelemetry with @shravanipatil527 and I tried your suggestion but it didn't work. The option was "Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins".

It is related to the changes to mirror the logs locally at https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/opentelemetry-2.10.0 Now you have to specify the property otel.logs.mirror_to_disk=true in advanced settings. So, the option to store logs locally from the backend Elastic Stack does nothing locally now. The logic was removed.

Enabling the local mirror you lose the link to the logs in the Console output, you still send the logs to Elasticsearch but you have to find them manually.

kuisathaverat commented 1 year ago

The option was "Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins".

That option is what it is broken, it no longer retrieves the logs from Elasticsearch, the only option Store pipeline logs In Elastic and visualize logs in Elastic, then enable otel.logs.mirror_to_disk=true. This has the issue that you lost the link to Kibana to see the logs. If you do not enable otel.logs.mirror_to_disk=true you will only see the link to Kibana to view the logs.

Geni2 commented 1 year ago

The option was "Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins".

That option is what it is broken, it no longer retrieves the logs from Elasticsearch, the only option Store pipeline logs In Elastic and visualize logs in Elastic, then enable otel.logs.mirror_to_disk=true. This has the issue that you lost the link to Kibana to see the logs. If you do not enable otel.logs.mirror_to_disk=true you will only see the link to Kibana to view the logs.

These are the options that I have. Also I added this otel.logs.mirror_to_disk=true in the place that you have mentioned. But still can't see the logs on the console output. Screenshot 2023-10-05 111354

kuisathaverat commented 1 year ago

Weird, I have tested it with Jenkins Core 2.401.3 and OpenTelemetry plugin 2.16

Geni2 commented 1 year ago

Weird, I have tested it with Jenkins Core 2.401.3 and OpenTelemetry plugin 2.16

To add more context about weird, during the pipeline run until it finished I could see the logs in the console output after the pipeline finished and you exit that page or do a page reload it's empty. This also affects all pipeline runs before enabling this option.

I will see to update the OpenTelemetry plugin the current version is 2.12.0

Screenshot 2023-10-05 133806

We have many Jenkins environments with the same version and use the same OpenTelemetry version. But only one has this problem which is really strange.

lucasmat7 commented 1 year ago

Picking up from @shravanipatil527 and @Geni2 , I've upgraded the plugin to the latest version (current 2.17) and still no luck. I've tried different combinations:

"Store pipeline logs In Elastic and visualize logs exclusively in Elastic" option + otel.logs.mirror_to_disk=true
"Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins" option
"Store pipeline logs In Elastic and visualize logs both in Elastic and through Jenkins" option + otel.logs.mirror_to_disk=true

with curl I was able to query elasticsearch search api from the jenkins controller node to retrieve records from the index:

I also observed something very interesting, when any other setting that is not Don't store pipeline logs in Elastic is selected, the following behavior occurs:

you run a build
if you check in the jenkins controller filesystem, the build log is there, the pipeline output is present (path example: /var/lib/jenkins/jobs/MyJobName/builds/60563/log
You go to the UI and click on the build > console output, the error that we are discussing is present
If now you check the filesystem again, the log file is empty

Also, it does not show the button "View logs in Elastic Observability" as it should

I'll continue investigating it...

lucasmat7 commented 1 year ago

I've found the problem... We had a plugin called "Delete logs" enabled, and as soon as it got disabled everything worked as expected

So I don't think it's a problem with this plugin, therefore we can close the Issue

kuisathaverat commented 10 months ago

in some cases, the cause is the index pattern, it needs to be logs-apm.app-*,.ds-logs-apm.app* instead of only logs-apm.app-* on Elastic Stack 8+

jenkinsci / opentelemetry-plugin