Limit the agent history

mawinter69 commented 5 months ago

What feature do you want to see added?

When an agent is heavily used by many jobs that are fast and run frequently the history can easily reach thousands of runs. This can lead to noticable delays when trying to open the details of a run to show all the matching steps (might also be caused by the YUI stuff). By limiting the number of entries this can reduce load on the backend and speed up the UI. Paging might also be a solution in case the ui should be fast but we need access to older entries.

Upstream changes

No response

Are you interested in contributing this feature?

No response

alwaysharsha commented 3 months ago

Any update on limiting the agent history or paginate the history?

Waschndolos commented 3 months ago

That would help us as well. We're having a pretty big Jenkins with > 2500 Jobs and ~300 Agents and the system is pretty much unusable when this plugin is installed (Guess it's expected to be due to the size).

Having a configuration option to limit the data could help here. I think the plugin itself is really great but with big systems it's unfortunatelly unusable for us right now (but we want it :) ).

grant-singleton-nz commented 1 month ago

I wouldn't call this an enhancement, I'd call it a bug. It completely takes down Jenkins for us.

If someone clicks on the extended build history for an agent, Jenkins becomes unresponsive a few minutes later and requires a manual restart.

We've disabled the plug-in for now, but its very useful so hoping this issue can get some attention soon too.

mawinter69 commented 1 month ago

Hmm, it's strange that the whole Jenkins becomes unusable. On first access of the history of an agent a background thread starts, that will collect the data of the builds that started before Jenkins was started. The reason is that finding out on which agents a pipeline was run is not straightforward like it is for freestyle jobs. And doing that each time one accesses the history of an agent would take too long and would produce even more load when this is done for several agents in parallel. What I could imagine is that you have a memory problem and the JVM runs GC very frequently. If this is the case you should look into optimizing the GC settings and/or increase the memory. You might want to check <root>/manage/systemInfo and then Memory Usage if you run out of heap space. Jenkins has lazy run loading and without the plugin it might mean that many runs never get loaded into memory. The plugin will basically force loading of old runs (though this should also happen if you click on the Build History link of an agent).

I know that the UI with the history can become laggy when you have many runs to show and the intention of this issue is to fix this lagginess.

ChristianMa97 commented 1 month ago

We also need this functionality, so I'm currently working on a fix. This will likely take a few days to complete due to some larger changes required, particularly those involving serialization and other related aspects.

I'll keep you updated on the progress.

grant-singleton-nz commented 1 month ago

@mawinter69 Turns out it is a GC problem we're having, Jenkins is running happily using ~3GB ram, then someone clicks on the extended job history and within seconds heap usage goes to ~5GB, the extra 2GB being in the old generation. We've given Jenkins a max heap of 5GB so it goes into a frequent GC loop and never recovers.

ChristianMa97 commented 3 weeks ago

I've made some promising changes that I believe address the issue, but I still need to conduct extensive testing to ensure everything works as expected. Once the testing is complete and everything checks out, I'll proceed with submitting a pull request.

ChristianMa97 commented 3 weeks ago

The testing for the memory optimization changes is looking promising. We have already deployed the updated version on some smaller production systems, and everything seems stable so far. However, we can only test it on a larger Jenkins instance later.

If anyone is interested in testing the fix on their systems, you can pull from this fork and branch: https://github.com/ChristianMa97/pipeline-agent-build-history-plugin/tree/bugfix/prevent-oom. After cloning, you can run mvn install to build the plugin.

Once the build is complete, you'll find the .hpi file in the target/ folder, which can be manually deployed to your Jenkins instance by uploading it via Manage Jenkins > Manage Plugins > Advanced > Upload Plugin.

Brief Overview of Changes:

Build metadata is now serialized, and only the objects being displayed at any time are held in memory, reducing the overall memory footprint.
Pagination has been added, allowing users to view builds in a paginated manner, ensuring efficient memory usage even with large build histories.
The configuration page allows you to define the storage directory, default entries per page, and default sorting settings.

mawinter69 commented 3 weeks ago

Would you mind opening a PR now (If it's not ready yet put it to draft). This will trigger a central build and allows to download the .hpi file from jenkins.io without the need to build it locally.

jenkinsci / pipeline-agent-build-history-plugin