jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
95 stars 49 forks source link

Too many files opened #716

Closed DuMaM closed 11 months ago

DuMaM commented 11 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.375.4 OS: Linux - 3.10.0-1160.45.1.el7.x86_64 Java: 11.0.16 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.20.0 Parameterized-Remote-Trigger:3.2.0 allure-jenkins-plugin:2.30.3 amazon-ecr:1.114.vfd22430621f5 analysis-model-api:10.23.1 ansicolor:1.0.4 antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 audit-trail:333.vb_e1b_b_0f1238c authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.529-406.vdeff15e5817d aws-java-sdk-cloudformation:1.12.529-406.vdeff15e5817d aws-java-sdk-codebuild:1.12.529-406.vdeff15e5817d aws-java-sdk-ec2:1.12.529-406.vdeff15e5817d aws-java-sdk-ecr:1.12.529-406.vdeff15e5817d aws-java-sdk-ecs:1.12.529-406.vdeff15e5817d aws-java-sdk-efs:1.12.529-406.vdeff15e5817d aws-java-sdk-elasticbeanstalk:1.12.529-406.vdeff15e5817d aws-java-sdk-iam:1.12.529-406.vdeff15e5817d aws-java-sdk-kinesis:1.12.529-406.vdeff15e5817d aws-java-sdk-logs:1.12.529-406.vdeff15e5817d aws-java-sdk-minimal:1.12.529-406.vdeff15e5817d aws-java-sdk-secretsmanager:1.12.529-406.vdeff15e5817d aws-java-sdk-sns:1.12.529-406.vdeff15e5817d aws-java-sdk-sqs:1.12.529-406.vdeff15e5817d aws-java-sdk-ssm:1.12.529-406.vdeff15e5817d basic-branch-build-strategies:81.v05e333931c7d bootstrap5-api:5.2.1-3 bouncycastle-api:2.29 branch-api:2.1105.v472604208c55 build-failure-analyzer:2.4.2 build-name-setter:2.3.0 build-timeout:1.31 build-token-root:151.va_e52fe3215fc caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:1.8.1 cloudbees-disk-usage-simple:182.v62ca_0c992a_f3 cloudbees-folder:6.815.v0dd5a_cb_40e0e cobertura:1.17 code-coverage-api:3.5.0 command-launcher:107.v773860566e2e commons-httpclient3-api:3.1-3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.10.0-68.v0d0b_c439292b_ config-file-provider:953.v0432a_802e4d2 configuration-as-code:1625.v27444588cc3d confluence-publisher:156.vf3597ca_9cf27 copyartifact:714.v28a_34f8c563f credentials:1224.vc23ca_a_9a_2cb_0 credentials-binding:626.v8d9034b_8ea_cc crowd2:4.0.0 cucumber-reports:5.7.6 dark-theme:336.v02165cd8c2ee data-tables-api:1.12.1-4 declarative-pipeline-migration-assistant:1.6.1 declarative-pipeline-migration-assistant-api:1.6.1 display-url-api:2.3.9 docker-commons:419.v8e3cd84ef49c docker-workflow:572.v950f58993843 durable-task:523.va_a_22cf15d5e0 ec2:1609.v53b_02a_b_9e52d echarts-api:5.4.0-1 email-ext:2.99 envinject:2.908.v66a_774b_31d93 envinject-api:1.199.v3ce31253ed13 extended-choice-parameter:376.v2e02857547b_a_ extended-read-permission:3.2 favorite:2.4.3 file-leak-detector:1.12 file-operations:1.11 font-awesome-api:6.2.1-1 forensics-api:1.17.0 ghprb:1.42.2 git:5.0.2 git-client:4.3.0 git-parameter:0.9.19 github:1.37.2 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1728.v859147241f49 github-checks:554.vb_ee03a_000f65 gradle:2.8.2 groovy:453.vcdb_a_c5c99890 h2-api:1.4.199 htmlpublisher:1.31 http_request:1.18 ignore-committer-strategy:1.0.4 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jacoco:3.3.4 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 jersey2-api:2.40-1 jira:3.11 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.82 jobConfigHistory:1212.vd4470d08ff12 jquery3-api:3.6.1-2 jsch:0.2.8-65.v052c39de79b_2 junit:1202.v79a_986785076 kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 lockable-resources:1131.vb_7c3d377e723 mailer:457.v3f72cb_e015e5 mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:150.vf80d33113e80 matrix-auth:3.1.9 matrix-project:808.v5a_b_5f56d6966 maven-plugin:3.22 metrics:4.2.18-439.v86a_20b_a_8318b_ mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ monitoring:1.95.0 nexus-artifact-uploader:2.14 node-iterator-api:49.v58a_8b_35f8363 nodejs:1.6.0.1 nodelabelparameter:1.11.0 oauth-credentials:0.646.v02b_66dc03d2e oic-auth:2.6 okhttp-api:4.11.0-145.vcb_8de402ef81 opentelemetry:2.16.0 pam-auth:1.10 parameter-separator:1.3 parameterized-trigger:2.45 pipeline-aws:1.43 pipeline-build-step:491.v1fec530da_858 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-graph-view:198.v0844db_6ca_554 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-maven:1342.vfc697b_789147 pipeline-maven-api:1342.vfc697b_789147 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2133.ve46a_6113dfc3 pipeline-model-definition:2.2133.ve46a_6113dfc3 pipeline-model-extensions:2.2133.ve46a_6113dfc3 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2133.ve46a_6113dfc3 pipeline-stage-view:2.33 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-usage-plugin:4.2 plugin-util-api:2.20.0 popper2-api:2.11.6-2 postbuildscript:3.2.0-460.va_fda_0fa_26720 prism-api:1.29.0-2 pubsub-light:1.17 purge-build-queue-plugin:88.v23b_97b_f2c7a_d purge-job-history:1.6 rebuild:320.v5a_0933a_e7d61 resource-disposer:0.23 s3:0.12.3445.vda_704535b_5a_d scm-api:676.v886669a_199a_a_ script-security:1269.v639888f5e366 slack:684.v833089650554 snakeyaml-api:2.2-111.vc6598e30cc65 sonar:2.15 sse-gateway:1.26 ssh-agent:333.v878b_53c89511 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.312.v1c601b_c83b_0e strict-crumb-issuer:2.1.1 structs:325.vcb_307d2a_2782 subversion:2.17.3 testng-plugin:789.vfc860d1de85a_ theme-manager:193.vcef22f6c5f2b_ throttle-concurrents:2.14 timestamper:1.26 token-macro:384.vf35b_f26814ec trilead-api:2.84.v72119de229b_7 uno-choice:2.7.2 validating-string-parameter:2.8 variant:60.v7290fc0eb_b_cd versioncolumn:145.va_e3ca_f8a_a_d23 view-job-filters:2.3 warnings-ng:9.23.1 workflow-aggregator:596.v8c21c963d92d workflow-api:1281.vca_5fddb_3fceb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3673.v5b_dd74276262 workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1326.ve643e00e9220 workflow-multibranch:746.v05814d19c001 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Open telementry plugin do not close it's sockets or files.

Screenshot 2023-09-26 at 16 02 42

Reproduction steps

  1. Add this config:
openTelemetry:
    authentication:
      otlpHeaderAuthentication:
        headerName: "api-key"
        headerValueId: "<some-value>"
    configurationProperties: |-
      otel.logs.exporter=otlp
      otel.logs.mirror_to_disk=true
    disabledResourceProviders: "io.opentelemetry.instrumentation.resources.ProcessResourceProvider"
    endpoint: "https://otlp.nr-data.net:4317"
    exportOtelConfigurationAsEnvironmentVariables: false
    ignoredSteps: "dir,echo,isUnix,pwd,properties"
    serviceName: "some app endpoint"
    serviceNamespace: "some name"
  1. Observe number of files used by jenkins

Expected Results

The number of opened descriptors should stay constant after some time.

Actual Results

After enabling:

  otel.logs.exporter=otlp
  otel.logs.mirror_to_disk=true

Number of opened files sky rocket

Anything else?

No response

cyrille-leclerc commented 11 months ago

Probably a duplicate of

Sorry for the inconvenience, can you please test our latest version:

DuMaM commented 11 months ago

Yes i will try it out, in next week. Thanks for quick response :)

DuMaM commented 11 months ago

@cyrille-leclerc I tested this out on a smaller dev instance and I don't see this type of behavior anymore.

DuMaM commented 11 months ago

Ok, i confirming that it works.