search

NixOS / hydra

Hydra, the Nix-based continuous build system
http://nixos.org/hydra
GNU General Public License v3.0
1.17k stars 300 forks source link

Declarative jobsets plugin is always enabled, causing buildcompleted events to always get fetched from the DB #1337

Open delroth opened 8 months ago

delroth commented 8 months ago

Describe the bug The plugin system is optimized so that it doesn't load a full event from the DB if no plugin is interested in it. Because the declarativejobset plugin says it's interested in everything, all the buildcompleted events are always loaded from the DB by hydra-notify, which is particularly annoying because each of those events are quite heavy to load (many DB queries for each event).

On hydra.nixos.org, this is causing hydra-notify to constantly stream 750Mbps of (seemingly) eval error text from PostgreSQL just to drop them on the floor and do nothing. Not very efficient.

To Reproduce n/a

Expected behavior The declarative jobset system isn't a drain on Hydra's resources when not in use.

Screenshots n/a

Hydra Server:

hydra.nixos.org.

delroth commented 8 months ago

Note that working around this issue surfaced a second problem: if no plugin accepts a buildcompleted event, it's never marked as handled in the database because updating notificationpendingsince is dependent on execute being called at least once (which induces ->load being called at least once, which is not wanted).

mweinelt commented 4 months ago

Lately the hydra-notify service has been sitting on rhea with ~45 GB memory usage, which is a problem since it impacts hydras ability to successfully carry out evaluations.