Memory leak after upgrading to 2.0.10

michaelbra commented 3 years ago

Version report

Jenkins and plugins versions report:

Jenkins: 2.277.4
OS: Linux - 4.4.1-1.el7.elrepo.x86_64
---
jquery-detached:1.2.1
ace-editor:1.1
momentjs:1.1.1
workflow-aggregator:2.6
pipeline-github-lib:1.0
mapdb-api:1.0.9.0
build-monitor-plugin:1.12+build.201809061734
build-pipeline-plugin:1.5.8
clang-scanbuild:1.8
docker-custom-build-environment:1.7.3
docker-slaves:1.0.7
DotCi:2.40.00
external-monitor-job:1.7
global-build-stats:1.5
slack-uploader:1.7
text-file-operations:1.3.2
ruby-runtime:0.12
ci-skip:0.0.2
pipeline-graph-analysis:1.10
jaxb:2.3.0.1
cppcheck:1.25
jquery:1.12.4-1
test-results-analyzer:0.3.5
git-server:1.9
embeddable-build-status:2.0.3
cmakebuilder:2.6.3
envinject-api:1.7
gradle:1.36
pam-auth:1.6
jsch:0.1.55.2
ant:1.11
plain-credentials:1.7
ssh-credentials:1.18.1
cobertura:1.16
job-restrictions:0.8
pipeline-stage-step:2.5
extended-read-permission:3.2
cloudbees-disk-usage-simple:0.10
authentication-tokens:1.4
okhttp-api:3.14.9
antisamy-markup-formatter:2.1
docker-commons:1.17
build-timeout:1.20
pipeline-build-step:2.13
snakeyaml-api:1.27.0
matrix-project:1.18
logstash:2.4.0
workflow-job:2.40
javadoc:1.6
workflow-step-api:2.23
credentials-binding:1.24
git-parameter:0.9.13
pipeline-input-step:2.12
scm-api:2.6.4
groovy:2.3
lockable-resources:2.10
ansible:1.1
htmlpublisher:1.25
handy-uri-templates-2-api:2.1.8-1.0
sse-gateway:1.24
blueocean-autofavorite:1.2.4
postbuild-task:1.9
dtkit-api:3.0.0
display-url-api:2.3.4
pipeline-rest-api:2.19
pipeline-stage-view:2.19
trilead-api:1.0.13
run-condition:1.5
conditional-buildstep:1.4.1
cloudbees-folder:6.15
workflow-basic-steps:2.23
rebuild:1.32
workflow-support:3.8
popper-api:1.16.1-2
ws-cleanup:0.39
apache-httpcomponents-client-4-api:4.5.13-1.0
workflow-scm-step:2.12
script-security:1.76
workflow-api:2.42
pipeline-model-api:1.8.4
pipeline-model-extensions:1.8.4
pipeline-stage-tags-metadata:1.8.4
pipeline-model-definition:1.8.4
metrics:4.0.2.7
copyartifact:1.46
variant:1.4
data-tables-api:1.10.23-3
forensics-api:1.0.0
resource-disposer:0.15
pipeline-milestone-step:1.3.2
handlebars:3.0.8
bouncycastle-api:2.20
throttle-concurrents:2.2
email-ext:2.82
junit:1.49
jjwt-api:0.11.2-9.c8b45b8bb173
github-api:1.123
docker-workflow:1.26
code-coverage-api:1.3.2
jdk-tool:1.5
mailer:1.34
github:1.33.1
ssh-slaves:1.31.5
blueocean-display-url:2.4.1
build-with-parameters:1.5.1
gitlab-plugin:1.5.20
favorite:2.3.3
envinject:2.4.0
maven-plugin:3.10
token-macro:2.15
parameterized-trigger:2.40
docker-build-publish:1.3.3
blueocean-commons:1.24.6
blueocean-jwt:1.24.6
bootstrap4-api:4.6.0-3
monitoring:1.87.0
blueocean-rest:1.24.6
checks-api:1.7.0
blueocean-jira:1.24.6
jquery3-api:3.6.0-1
jenkins-design-language:1.24.6
blueocean-core-js:1.24.6
blueocean-pipeline-scm-api:1.24.6
blueocean-web:1.24.6
blueocean-rest-impl:1.24.6
blueocean-pipeline-api-impl:1.24.6
blueocean-github-pipeline:1.24.6
blueocean-git-pipeline:1.24.6
blueocean-config:1.24.6
blueocean-bitbucket-pipeline:1.24.6
blueocean-dashboard:1.24.6
blueocean-personalization:1.24.6
blueocean-events:1.24.6
blueocean-pipeline-editor:1.24.6
blueocean-i18n:1.24.6
blueocean:1.24.6
git-client:3.7.1
git:4.7.1
windows-slaves:1.8
command-launcher:1.6
cloudbees-bitbucket-branch-source:2.9.8
subversion:2.14.1
timestamper:1.13
workflow-cps-global-lib:2.19
branch-api:2.6.4
plugin-util-api:2.2.0
jackson2-api:2.12.3
jira:3.3
font-awesome-api:5.15.3-2
xunit:3.0.2
durable-task:1.36
structs:1.23
analysis-model-api:10.2.5
mercurial:2.15
pubsub-light:1.14
echarts-api:5.1.0-2
warnings-ng:9.1.0
credentials:2.4.1
workflow-durable-task-step:2.39
matrix-auth:2.6.7
workflow-cps:2.92
github-branch-source:2.10.4
build-name-setter:2.2.0
ansicolor:1.0.0
prometheus:2.0.10
ldap:2.7
workflow-multibranch:2.24
slack:2.48

What Operating System are you using (both controller, and any agents involved in the problem)?

CentOS 7 running docker version 1.13.1. Jenkins LTS 2.227.4 docker image from https://hub.docker.com/r/jenkins/jenkins

Reproduction steps

Upgrading Jenkins from 2.277.1 to 2.277.4
Upgrading several plugins at the same time, incl. prometheus plugin from 2.0.9 to 2.0.10
After the upgrade docker logs are filled with these entries:

2021-05-15 03:21:43.417+0000 [id=30288] WARNING hudson.FilePath#_getChannelForSerialization: A FilePath object is being serialized when it should not be, indicating a bug in a plugin. See https://jenkins.io/redirect/filepath-serialization for details.
java.io.NotSerializableException: The calling thread Thread[prometheus_async_worker thread,5,main] has no associated channel. The current object null is interface org.jenkinsci.remoting.SerializableOnlyOverRemoting, but it is likely being serialized/deserialized without the channel
    at org.jenkinsci.remoting.SerializableOnlyOverRemoting.getChannelForSerialization(SerializableOnlyOverRemoting.java:67)
    at hudson.FilePath._getChannelForSerialization(FilePath.java:3265)
    at hudson.FilePath.readObject(FilePath.java:3273)
    at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.thoughtworks.xstream.core.util.SerializationMembers.callReadObject(SerializationMembers.java:132)
    at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doUnmarshal(SerializableConverter.java:443)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:277)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:439)
    at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:341)
    at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:280)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
    at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
    at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:439)
    at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:341)
    at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:280)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1429)
    at hudson.util.XStream2.unmarshal(XStream2.java:161)
    at hudson.util.XStream2.unmarshal(XStream2.java:132)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1409)
    at hudson.XmlFile.unmarshal(XmlFile.java:180)
    at hudson.XmlFile.unmarshal(XmlFile.java:163)
    at hudson.model.Run.reload(Run.java:364)
    at hudson.model.Run.<init>(Run.java:352)
    at hudson.model.AbstractBuild.<init>(AbstractBuild.java:174)
    at hudson.model.Build.<init>(Build.java:104)
    at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38)
    at sun.reflect.GeneratedConstructorAccessor747.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:166)
    at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:143)
    at hudson.model.RunMap.retrieve(RunMap.java:224)
    at hudson.model.RunMap.retrieve(RunMap.java:57)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346)
    at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:368)
    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:195)
    at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104)
    at org.jenkinsci.plugins.prometheus.JobCollector.appendJobMetrics(JobCollector.java:316)
    at org.jenkinsci.plugins.prometheus.JobCollector.lambda$collect$0(JobCollector.java:204)
    at org.jenkinsci.plugins.prometheus.util.Jobs.forEachJob(Jobs.java:14)
    at org.jenkinsci.plugins.prometheus.JobCollector.collect(JobCollector.java:197)
    at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190)
    at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223)
    at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144)
    at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
    at org.jenkinsci.plugins.prometheus.service.DefaultPrometheusMetrics.collectMetrics(DefaultPrometheusMetrics.java:54)
    at org.jenkinsci.plugins.prometheus.service.PrometheusAsyncWorker.execute(PrometheusAsyncWorker.java:41)
    at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
    at java.lang.Thread.run(Thread.java:748)

Results

Jenkins got unresponsive for 1,5 days (over weekend) and the logs got spammed with tons of above described entries. JVM memory usage went up a lot. Garbage collector was able to free memory, but building up quickly afterwards.

Rolling back to 2.0.9 fixed the issue.

michaelbra commented 3 years ago

Could it be related to #183 ?

dzmitrytarasevich commented 3 years ago

Hi guys. I faced with the same issue. I'm using Prometheus metrics 2.0.7, Jenkins 2.277.4 Will update it to 2.0.9 and let you know the result.

dzmitrytarasevich commented 3 years ago

After upgrading to version 2.0.9 the issue stay there. But after upgrading to version 2.0.10 I have no more java.io.NotSerializableException: The calling thread Thread[prometheus_async_worker thread,5,main] entries.

markjacksonfishing commented 3 years ago

Please use latest version

michaelbra commented 3 years ago

@markyjackson-taulia Thanks, but there is no 2.0.11 available yet on Jenkins: https://plugins.jenkins.io/prometheus/#releases

jenkinsci / prometheus-plugin