[JENKINS-37570] Jenkins starts to response super slow after creating 400 branch jobs from Multi-branch-project

[timja]

Right after doing branch-indexing to create 400 branch jobs,
Jenkins starts to response super slow when open any pages (job page, manage jenkins page, etc).

And after delete the Multi-branch job of 400 branch jobs,
everything is back to normal.

From the log,
I don't see any out of memory error.
But I do see the Java process has high %CPU (400~500) with about 1.9G RES memory.

---

Originally reported by totoroliu, imported from: Jenkins starts to response super slow after creating 400 branch jobs from Multi-branch-project

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

danielbeck:

Would probably help if you did this: https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump

[timja]

totoroliu:

This is the threddump I did using method1 (from Jenkins Web UI)
when I click the multi-branch project that contains 300+ branch jobs.

[timja]

splashnenen:

Hi,

I'm facing also a similar problem. We have around 60 multibranch pipelines jobs (not so many) and all 'branch indexing' are run at the same time. At this moment, jenkins becomes really slow and indexing might last 3 or 4 minutes instead of a few seconds, taken 1 by 1.

There is no specific hardware limitation on the jenkins master and GIT server is also well sized. The configuration "Periodically if not otherwise run", if set to '5 minutes' for example has not the expected impact. Indeed the load is still the same because it starts all indexing operations at the same time, every 5 minutes. it seems that this configuration is based on absolute time so there is no repartition of load using this.

Thanks for your help.

 

 

[timja]

splashnenen:

i increase the criticity because our Jenkins is unusable when too many branch indexing operations are triggered, and this is not configurable so it directly depends on multibranch jobs count.

 

It would be really nice to find a solution for this.

[timja]

stephenconnolly:

concurrent branch indexing should be capped at 5 -> https://github.com/jenkinsci/cloudbees-folder-plugin/blob/806efc0db12cd6cbcc90f0acbb0ea1ab7b942259/src/main/java/com/cloudbees/hudson/plugins/folder/computed/ThrottleComputationQueueTaskDispatcher.java#L49

I wonder is this the pipeline jobs themselves as a pipeline job can start executing but remain "hidden" while the sub-tasks are queued

[timja]

stephenconnolly:

splashnenen keep in mind that you should be using event push notification and your periodically should be ideally every 8h or longer (depends on how often you want to clean out orphaned branches)

[timja]

stephenconnolly:

https://issues.jenkins-ci.org/browse/JENKINS-43424 should address the coordinated triggering for short intervals

[timja]

stephenconnolly:

Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA