search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-57216] Upgrade to v2.173 Made Everything Slow #4391

Open timja opened 5 years ago

timja commented 5 years ago

We were running version 2.156 and decided to upgrade to v2.173. 
After doing so everything was running slow on the Jenkins UI. Just opening up the dashboard, Job log, Job configuration pages took up to 20 seconds to load. 

We checked the CPU and Memory Usage and nothing was wrong there the machine was fine. 

We needed to Downgrade back to v2.156 and now everything is working back to normal

Originally reported by yossisilb, imported from: Upgrade to v2.173 Made Everything Slow
  • status: Open
  • priority: Minor
  • resolution: Unresolved
  • imported: 2022/01/10
timja commented 5 years ago

oleg_nenashev:

Would it be possible to profile instance or to provide information to make the case reproducible? I do not see immediate root cause in the fixes

timja commented 5 years ago

yossisilb:

oleg_nenashev What information do you need exactly? 

What do you mean about the instance? 

timja commented 5 years ago

batmat:

Would be also great if you can identify more precisely which exact Jenkins release starts to behave badly for you. I.e. test every release between v2.156 and 2.173 to identify a precise version. (note: you can likely do it by dichotomy to not go through ALL releases)

timja commented 5 years ago

tsaraja:

We used 2.138.3 and now updated to 2.176.2. Both of them extremely slow when hitting 'configure' for any particular job we have. Web inspector in Chrome said that it was idle for 12 seconds and 8 seconds it presented the configurations...among other things rendered.

When running htop simultaneously in the host, I could see that the first CPU hit 120% and the second one over 9% during those 12 seconds Chrome was idle, and the process causing it was Jenkins related. So apparently it really takes 12 seconds for Jenkins to load configurations.

But it's not Jenkins version related. We also have 2.176.1 running with vast amounts of configurations. There loading an example configuration takes 5 seconds in total, of which it is idle 1.8 seconds, scripts for 1.7s, renders 800ms etc. These configurations aren't identical, so in our case it might be that we need to compare them. But be the reason for the difference whatever, loading configurations for several seconds is very slow.