Prometheus API return blank page

jenkinsci / prometheus-plugin

Jenkins Prometheus Plugin

https://plugins.jenkins.io/prometheus/

Apache License 2.0

184 stars 152 forks source link

Prometheus API return blank page #82

Closed JingangLi closed 5 years ago

JingangLi commented 5 years ago

Hi,

I try to use Prometheus plugin in our jenkins setup. As the whole Jenkins is setup by another team, and I just install Prometheus plugin, so I have limited control on the system.

My prometheus plugin version is 2.0.0, metrics plugin is 4.0.2.2 and my jenkins is Jenkins 2.89.2. And now the prometheus API just return blank page. I think the problem is different with other tickets here. As I try to setup the log with org.jenkinsci.plugins.prometheus. Any thing prints out, so I guess the prometheus plugin doesn't execute.

Do you have some idea I can trouble shooting this problem.

BTW the metrics API works well when I generete Key for access it.

Rgds, Jingang Li

JingangLi commented 5 years ago

Hi @markyjackson-taulia ,

Do you have some guide, I can trouble shooting the problem by myself? I think to know where is the problem which is already very helpful for us.

Many thanks.

Rgds, Jingang Li

markjacksonfishing commented 5 years ago

Are you able to view the Jenkins logs?

JingangLi commented 5 years ago

Hi @markyjackson-taulia ,

Thanks for replying me. I can see the log of Jenkins on general logger. But there isn`t content for Prometheus plugin.

And I also create log on org.jenkinsci.plugins.prometheus, then no log is collected. That causes me thought the plugin is executed.

Rgds, Jingang Li

konrad-a commented 5 years ago

I have the same issue. There are no Prometheus entries in my Master logs. curl -v https://my-jenkins/prometheus/ returns 302 Found

curl -v https://my-jenkins/prometheus returns 200 OK

If I change my settings to use authentication for the endpoint the curl returns 4xx as expected (so the plugin kind of "works") The result page is always empty.

I tried upgrading from 1.2.2 to 2.0.0 and the issue is still the same.

BTW: where I can find 1.2.3 and 1.2.4 release binaries? I cannot see them here: https://updates.jenkins-ci.org/download/plugins/prometheus/

markjacksonfishing commented 5 years ago

@konrad-a oh that is interesting the binaries are not there. Let me get that figured out with artifactory Thanks for bringing that up. Can you do me a favor and open a separate issue for that so I can track these individually?

markjacksonfishing commented 5 years ago

Can you both verify that your prometheus is showing the Jenkins target as 1/1 up, like this:

konrada commented 5 years ago

@konrad-a oh that is interesting the binaries are not there. Let me get that figured out with artifactory Thanks for bringing that up. Can you do me a favor and open a separate issue for that so I can track these individually? Opened #89

guillaumeger commented 5 years ago

We have the same problem:

bash-4.2$ curl -v localhost:8080/prometheus/
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /prometheus/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8080
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Feb 2019 20:24:30 GMT
< X-Content-Type-Options: nosniff
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Content-Length: 0
< Server: Jetty(9.4.z-SNAPSHOT)
< 
* Connection #0 to host localhost left intact

and

bash-4.2$ curl -v localhost:8080/prometheus 
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /prometheus HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8080
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Mon, 04 Feb 2019 20:25:03 GMT
< X-Content-Type-Options: nosniff
< Location: http://localhost:8080/prometheus/
< Content-Length: 0
< Server: Jetty(9.4.z-SNAPSHOT)
< 
* Connection #0 to host localhost left intact

markjacksonfishing commented 5 years ago

@guillaumeger Where are you running your Jenkins instance? Is it local? Would it be helpful if I setup a time I could help walk you through the setup?

guillaumeger commented 5 years ago

@markyjackson-taulia It is running in kubernetes in aws. Im a executing the curl command from within the pod. The manifest looks like this:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: jenkins
  name: jenkinsmaster
  labels:
    app: jenkinsmaster
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkinsmaster
      tier: backend
  template:
    metadata:
      labels:
        app: jenkinsmaster
        tier: backend
    spec:
      containers:
      - name: jenkinsmaster
        image: <redacted>/nuglif/jenkinsmaster:v2.147
        resources:
          requests:
            memory: 2500M
            cpu: 250m
          limits:
            memory: 3500M
            cpu: 500m
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: jenkins-home
          mountPath: "/var/jenkins_home"
        env:
        - name: JAVA_OPTS
          value: "-Dfile.encoding=UTF8 -Xmx2048m -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Dhudson.model.DirectoryBrowserSupport.CSP= -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/Montreal -Dpermissive-script-security.enabled=true -Dhudson.plugins.active_directory.ActiveDirectorySecurityRealm.forceLdaps=true"
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkinsmaster

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/proxy-body-size: 20m
    ingress.kubernetes.io/proxy-request-buffering: "off"
    kubernetes.io/ingress.class: nginx
  name: jenkinsmaster
  namespace: jenkins
spec:
  rules:
  - host: jenkins.<redacted>
    http:
      paths:
      - backend:
          serviceName: jenkinsmaster
          servicePort: 8080
        path: /

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkinsmaster
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

---

kind: Service
apiVersion: v1
metadata:
  name: jenkinsmaster
  namespace: jenkins
  labels:
    app: jenkinsmaster
    tier: backend
spec:
  selector:
    app: jenkinsmaster
    tier: backend
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
    name: ui
  - protocol: TCP
    port: 50000
    name: jnlp

The configuration of the plugin looks like that: screen shot 2019-02-05 at 09 23 49

I can successfully see metrics with https://<jenkins-url>/metrics/currentUser/metrics but https://<jenkins-url>/prometheus returns 200 with an empty page.

markjacksonfishing commented 5 years ago

Does your prometheus config have Jenkins configured in the scrape? Is your Jenkins target up in the Targets page of the Prometheus UI? You could also share your Prometheus configuration file and logs.

guillaumeger commented 5 years ago

screen shot 2019-02-05 at 16 01 51

When I exec into the prometheus server pod and do a wget on the jenkins pod ip:

/prometheus $ wget -S http://100.123.104.8:8080/prometheus
Connecting to 100.123.104.8:8080 (100.123.104.8:8080)
  HTTP/1.1 302 Found
  Connection: close
  Date: Tue, 05 Feb 2019 21:04:31 GMT
  X-Content-Type-Options: nosniff
  Location: http://100.123.104.8:8080/prometheus/
Connecting to 100.123.104.8:8080 (100.123.104.8:8080)
  HTTP/1.1 200 OK
  Connection: close
  Date: Tue, 05 Feb 2019 21:04:31 GMT
  X-Content-Type-Options: nosniff
  Content-Type: text/plain; version=0.0.4; charset=utf-8
  Content-Length: 0
  Server: Jetty(9.4.z-SNAPSHOT)

/prometheus $ ls -lah
total 20
drwxr-xr-x    2 nobody   nogroup     4.0K Feb  5 21:04 .
drwxr-xr-x    1 root     root        4.0K Dec 10 18:47 ..
-rw-r--r--    1 nobody   nogroup        0 Feb  5 21:04 prometheus
/prometheus $ cat prometheus 
/prometheus $

konrada commented 5 years ago

It got resolved for me after Master restart - the metrics where there under /prometheus/ endpoint. It reoccured after next restart today - the page is blank again. Can it be caused by some kind of initialisation / plugin load order issue?

guillaumeger commented 5 years ago

Following a restart, I now have this in jenkins log:

Feb 11, 2019 4:10:37 PM org.eclipse.jetty.server.handler.ContextHandler$Context log                                                                                                                                                                                                                                                                                      
WARNING: Error while serving http://100.123.104.7:8080/prometheus/                                                                                                                                                                                                                                                                                                       
java.lang.reflect.InvocationTargetException                                                                                                                                                                                                                                                                                                                              
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)                                                                                                                                                                                                                                                                                         
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)                                                                                                                                                                                                                                                                                                 
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)                                                                                                                                                                                                                                                                                 
        at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:408)                                                                                                                                                                                                                                                                                                 
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)                                                                                                                                                                                                                                                                                                       
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)                                                                                                                                                                                                                                                                                                          
        at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:384)                                                                                                                                                                                                                                                                                                 
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)                                                                                                                                                                                                                                                                                                       
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)                                                                                                                                                                                                                                                                                                          
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)                                                                                                                                                                                                                                                                                                          
        at org.kohsuke.stapler.Stapler.service(Stapler.java:238)                                                                                                                                                                                                                                                                                                         
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)                                                                                                                                                                                                                                                                                                  
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)                                                                                                                                                                                                                                                                                        
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)                                                                                                                                                                                                                                                                       
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)                                                                                                                                                                                                                                                                                      
        at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)                                                                                                                                                                                                                                                                  
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                                                                                                                                                                                                                                                                                      
        at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)                                                                                                                                                                                                                                                                             
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                                                                                                                                                                                                                                                                                      
        at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)                                                                                                                                                                                                                                                          
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                                                                                                                                                                                                                                                                                      
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)                                                                                                                                                                                                                                                                                           
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                                                                                                                                                                                                                                                                                      
        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)                                                                                                                                                                                                                                                                                   
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                                                                                                                                                                                                                                                                                      
        at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:124)                                                                                                                                                                                                                                                                    
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                                                                                                                                                                                                                                                                                      
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)                                                                                                                                                                                                                                                                                        
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)                                                                                                                                                                                                                                                                       
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)                                                                                                                                                                                                                                                                                                
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)                                                                                                                                                                                                                                                                       
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:503)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.StackOverflowError                                                                                                                                                                                                                                                                                                                                  
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1766)                                                                                                                                                                                                                                                 
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1169)                                                                                                                                                                                                                                                                       
        at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:165)                                                                                                                                                                                                                                                                                  
        at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:156)
        at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
        at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
        at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
        at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
        at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
        at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)

pavankurimilla commented 5 years ago

Hey folks, we also have the same issue with respect to prometheus plugin. is there any timeline for the fix? Few details about our setup: Jenkins Version: 2.95 Prometheus Metrics Plugin: 1.2.2 Metrics Plugin: 4.0.2.2 and Prometheus Targets say that the jenkins is up as well Appreciate it...

markjacksonfishing commented 5 years ago

https://github.com/jenkinsci/prometheus-plugin/issues/82#issuecomment-460688281

guillaumeger commented 5 years ago

Why is this closed? It's still not resolved...

markjacksonfishing commented 5 years ago

Because I am asking for the items in https://github.com/jenkinsci/prometheus-plugin/issues/82#issuecomment-460688281 but no one is giving that to me

pavankurimilla commented 5 years ago

i can add my details here... 1) yes, it is configured in the scrape 2) yes, jenkins target is up in the prometheus targets page 3) configmap entry:

job_name: 'jenkinsJob’ metrics_path: /prometheus scheme: http static_configs:
- targets:
  - jenkins.namespace basic_auth: username: someUser password: somepassword 3) Jenkins Version: 2.95 and is hosted in aws 4) Prometheus Plug in version is 1.2.2 4) Prometheus is also hosted in aws and in the same cluster as this jenkins instance and the image used is -- quay.io/prometheus/prometheus:v2.0.0

4) here is the log: error while serving http://jenkins.namespace/prometheus java.lang.StackOverflowError at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:75) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1393) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1076) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.getNode(SimpleXStreamFlowNodeStorage.java:104) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1770) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1169) at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:164) at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:155) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)

!!!!!!!!!!!!!!!!!!!!!!!! continued with the method call resulting in stackoverflow error

at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) Caused: java.lang.reflect.InvocationTargetException at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:397) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

markjacksonfishing commented 5 years ago

i can add my details here...

yes, it is configured in the scrape

yes, jenkins target is up in the prometheus targets page

configmap entry:

job_name: 'jenkinsJob’ metrics_path: /prometheus scheme: http static_configs:

targets:

jenkins.namespace basic_auth: username: someUser password: somepassword

here is the log: error while serving http://jenkins.namespace/prometheus java.lang.StackOverflowError at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:75) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1393) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1076) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.getNode(SimpleXStreamFlowNodeStorage.java:104) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1770) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1169) at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:164) at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:155) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)

!!!!!!!!!!!!!!!!!!!!!!!! continued with the method call resulting in stackoverflow error

at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48) Caused: java.lang.reflect.InvocationTargetException at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:397) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Can you tell me the version of Jenkins, prometheus plugin and prometheus you are using? Also can you tell me how you are hosting each of these (i.e. aws, local (if local, which the Mac, windows linux).

I will do any best to mimic your setup to see if I can reproduce the issue. As a note. I have set this up locally on my Mac as well as on an aws ec2 ubuntu instance and could not reproduce any of the problems described. My setup is:

Mac
Jenkins version 2.165
Plugin Version 2.0
Prometheus 2.7.1

pavankurimilla commented 5 years ago

just updated it. my hunch is -- prometheus endpoint is working fine till the scrape call comes in and it gets into an infinite loop and then never recovers. because, i see the job collector logs till the point of scrape call from prometheus and thats when it throws the above error and never recovers. i say it never recovers because i dont see any job collector logs after the exception. By the way, this behavior reproduced for both versions (1.2.2 and 2.0) versions of the Prometheus Metrics Plugin

guillaumeger commented 5 years ago

I posted a screenshot of my prometheus targets, jenkins is up. Also, I doubt it is related to the prometheus configuration since I get a blank page even when i just curl the endpoint, but I might be wrong.

Jenkins version: 2.147 Plugin version: 2.0.0 Prometheus version: 2.4.0 Metrics plugin: 4.0.2.2

Thanks for your help!

JingangLi commented 5 years ago

Hi @markyjackson-taulia,

Sorry for some late, I was in a vacation. In my case, it still doesn't work. And in same time, I don't think it related with Prometheus config. As I have two Jenkins setup, in one of them the API /jenkins/prometheus/ works well. In another setup, it doesn't work. And in the one doesn't work, I create Jenkins log for monitoring the plugin work, but the log is empty. So I am quite thinking the plugin doesn't work. Do you have some idea I can try to on my setup to know what happens?

Many thanks.

Rgds, Jingang Li

a-nldisr commented 5 years ago

Version using: Host: Centos 7, running Jenkins in docker container. Jenkins ver. 2.150.3 Plugins: Prometheus plugin: 2.0.0 Metrics plugin: 4.0.2.2 jUnit plugin: 1.26.1 Pipeline job: 2.31 Pipeline API: 2.33 bouncycastle API Plugin: 2.17 Command Agent Launcher: 1.2 JDK Tool: 1.1 JAXB: Not installed, not listed as required. Installed latest version just to exclude it as a cause. 2.3.0

Prometheus version 2.7.x (not relevant)

Issue: When we curl the jenkins server, with the /prometheus path, we get a 302.

curl -v http://jenkins.host:11040/prometheus
* About to connect() to jenkins.host port 11040 (#0)
*   Trying 10.0.7.15...
* Connected to jenkins.host (10.0.7.15) port 11040 (#0)
> GET /prometheus HTTP/1.1
> User-Agent: curl/7.29.0
> Host: jenkins.host:11040
> Accept: */*
>
< HTTP/1.1 302 Found
< Server: nginx/1.10.3
< Date: Tue, 05 Mar 2019 16:25:52 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Content-Type-Options: nosniff
< Location: http://jenkins.host:11040/prometheus/
<
* Connection #0 to host jenkins.host left intact

curl directly: curl -X GET jenkins.host:11040/prometheus empty response.

a-nldisr commented 5 years ago

Found out that when i connect the prometheus endpoint at the jenkins url thats configured in the jenkins settings i get metrics, this is an external url that is not used in our metrics gathering. We are using Prometheus with Marathon service discovery, this means that the ports exposed per service in our case by default get scraped for an metrics endpoint.

Changing the url to thats scraped internally: No metrics, empty response. @markyjackson-taulia Could this be fixed by adjustments in the jenkins plugin?

markjacksonfishing commented 5 years ago

@a-nldisr I will need to review that

a-nldisr commented 5 years ago

@markyjackson-taulia Found the real cause now: Does not work

curl -X GET jenkins.host:11040/prometheus

Does work:

curl -X GET jenkins.host:11040/prometheus/

The / at the end does the trick, maybe it helps others in this thread. Our load balancer to expose jenkins externally follows the location i assume.

daxroc commented 5 years ago

I'm seeing the same StackOverflow Assume this is to do with the number of Jobs and the number of historic builds.

Jenkins Version: 2.150.3


  at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:400)
  at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
  at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
  at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
  at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:482)
  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
  at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:458)
  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
  at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
  at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
  at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
  at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
  at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
  at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
  at org.eclipse.jetty.server.Server.handle(Server.java:503)
  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
  at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.StackOverflowError
  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.getNode(CpsFlowExecution.java:1794)
  at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getNode(CpsFlowExecution.java:1179)
  at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:165)
  at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:156)
  at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)
  at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:48)```

konrada commented 5 years ago

I can confirm that I can reproduce it and I have high number of jobs and a lot of historical builds

markjacksonfishing commented 5 years ago

If I got a meeting together for this issue with a video link where we can screen share and debug in realtime, would that help?

pavankurimilla commented 5 years ago

If I got a meeting together for this issue with a video link where we can screen share and debug in realtime, would that help?

Yes, I am in for it

peimanja commented 5 years ago

confirmed that curl -X GET jenkins.host:11040/prometheus/ works

NotTheEconomist commented 5 years ago

confirmed that curl -X GET jenkins.host:11040/prometheus/ works

Not in my case.

Last maintainer activity here was seven weeks ago -- is there still work being done on this issue?

FYI I'm hosting Jenkins locally on Windows.

markjacksonfishing commented 5 years ago

There is still work being done and I intend to be more active but this issue has languished due to a lack of concrete details. I did try to organize a meet early in this bug but got no respond

MatteoMori commented 5 years ago

Hello @markyjackson-taulia, apologies for disturbing you but we are having the same issue described above.I have installed the plugin on 3 Jenkins masters. The plugin works fine on 2 of them while on the 3rd one the endpoint does not return anything unfortunately (I get a 200 but no data).

The 3rd master is the busiest with quite a lot of jobs. These are the system details:

Jenkins: 2.138.4
Plugin version: 2.0.0
Deployment: Our Jenkins has been deployed as a container on top of Openshift 3.6
We are not using Prometheus. We are using Sysdig which is correctly collecting metrics from the other 2 masters.

As soon as we installed the plugin the memory utilisation increased drastically (I mean, I did expect that, so no problem here) but no data is returned at the endpoint.

We did restart the master. Still nothing.

I also checked in the log file but this is the only think I can see:

Jun 24, 2019 4:16:03 PM org.jenkinsci.plugins.prometheus.JobCollector getNamespace
INFO: The prometheus namespace is now [default]

Please do let me know if you need more info.

thanks, Matteo

markjacksonfishing commented 5 years ago

@MatteoMori Are the configuration and versions exactly the same?

MatteoMori commented 5 years ago

Hey Marky, thank you for replying so quickly. The configuration is in overall quite similar but unfortunately not 100% the same. The first 2 masters are based on the 3rd but have been deployed more recently and have a more recent version of Jenkins and some plugins.

To be honest, I just noticed that "Pipeline Utility Steps" is a little bit old (1.4.0) and I am wondering if that could case the issue. However, I did not see any request for upgrading any plugin when installing the Prometheus one, so I thought everything was fine.

Would you say that could be a possible reason for the issue? thanks again.

markjacksonfishing commented 5 years ago

@MatteoMori I think that is the possible reason

sesto commented 5 years ago

What is the reason for closing the issue? I am facing the same empty page issue after upgrading to the plugin version 2.0.6. Before the upgrade I have had this issue: https://github.com/jenkinsci/prometheus-plugin/issues/105 After the upgrade I have the blank page issue

wololock commented 5 years ago

Same issue on my side. The plugin version 2.0.6. Accessing ${JENKINS_URL}/prometheus/ produces a blank page. Jenkins Metrics plugin works without any issue.

Using curl:

$ curl -i "https://${JENKINS_HOST}/prometheus/"
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Wed, 16 Oct 2019 13:45:44 GMT
Content-Type: text/plain; version=0.0.4; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Content-Type-Options: nosniff
Cache-Control: must-revalidate,no-cache,no-store

karolgil commented 5 years ago

I've installed the plugin on 8 Jenkins instances so far - 1 of them with largest build history and many jobs returns blank page for /prometheus/ endpoint and I can see StackOverflowError (the one already posted in this issue) in Jenkins logs.

jw99324 commented 4 years ago

Was a solution ever found for this issue? I am facing the same issue on multiple Jenkins instances on Jenkins ver. 2.164.1 running Prometheus 2.0.6.

intelmatt commented 4 years ago

We are facing this issue also, on a server with a large number of jobs & builds:

2020-06-03 17:29:28.172+0000 [id=59980] SEVERE  h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (prometheus_async_worker thread/59980) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.StackOverflowError
    at java.base/java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(Native Method)
    at java.base/java.lang.StackStreamFactory$AbstractStackWalker.beginStackWalk(StackStreamFactory.java:370)
    at java.base/java.lang.StackStreamFactory$AbstractStackWalker.walk(StackStreamFactory.java:243)
    at java.base/java.lang.StackWalker.walk(StackWalker.java:498)
    at java.logging/java.util.logging.LogRecord$CallerFinder.get(LogRecord.java:709)
    at java.logging/java.util.logging.LogRecord.inferCaller(LogRecord.java:683)
    at java.logging/java.util.logging.LogRecord.getSourceMethodName(LogRecord.java:383)
    at io.jenkins.lib.support_log_formatter.SupportLogFormatter.format(SupportLogFormatter.java:71)
    at java.logging/java.util.logging.StreamHandler.publish(StreamHandler.java:199)
    at java.logging/java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:95)
    at java.logging/java.util.logging.Logger.log(Logger.java:979)
    at java.logging/java.util.logging.Logger.doLog(Logger.java:1006)
    at java.logging/java.util.logging.Logger.log(Logger.java:1117)
    at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:173)
    at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:155)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    at org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree(FlowNodes.java:49)
    ...

It looks like there is infinite recursion in org.jenkinsci.plugins.prometheus.util.FlowNodes.traverseTree. What can cause that infinite recursion? I can help to provide debug information as necessary.

cpitstick-argo commented 4 years ago

I re-wrote the FlowNodes class in 2.0.7 so that it doesn't have this bug anymore (Moved it to iterative). Build the plugin from source to eliminate the bug.

intelmatt commented 4 years ago

Thanks! Any guess when 2.0.7 will be released via the usual channel?

cpitstick-argo commented 4 years ago

Unfortunately that's out of my hands. I'm just a regular average-joe developer who felt I could contribute this fix as I saw the clear opportunity and I had the means. We later abandoned the plugin as it had scaling issues even after this fix was deployed, however, so be warned that this may not repair everything.

@markyjackson-taulia Is the guy for deploying the update broadly.

markjacksonfishing commented 4 years ago

Hey all. There have been infra issues that lasted through the weekend and into yesterday. MY hope is to get this out this week. With the world in disarray, I have been so out of it.

screighton13 commented 4 years ago

Any update on deploying the update?

look4regev commented 4 years ago

Installed 2.0.7 but still getting the blank response. Can you please advise? We need it so bad 🙏

p3t3rp3pp3r commented 4 years ago

Hi, We are facing this issue also in version 2.0.8