search

Nix-Security-WG / nix-security-tracker

Web service for managing information on vulnerabilities in software distributed through Nixpkgs
26 stars 4 forks source link

[Tracking Issue] Long-term performance of the security tracker #149

Open RaitoBezarius opened 8 months ago

RaitoBezarius commented 8 months ago

This is here to collect all the concerns about the long term performance of the security tracker.

A big part of the concern is with the disk space consumption rate and the query performance evolution over time.

Ideally, what we could do is maintain only a working set of data and archive old pages and freeze them so that we always work on one year or 3 months or 6 months worth of data to avoid the infinite growth of our dataset.

RaitoBezarius commented 8 months ago

I realized that .drv are probably going to stick around in the Nix store because we put GC roots on them. There is also a need to consume the .drv, unlink them and remove them explicitly from the store, don't let the GC procedure do it, it will be too slow and resource consuming.

This way, we ingest what we need out of those .drv and keep a reasonable operating Nix store size.

Now, we deployed the evaluation in production on spinning rust, I realize that the bottleneck is the nix-daemon performing some R/W, various metrics can be found here: http://sectracker.nixpkgs.lahfa.xyz:19999/#menu_systemd_web-security-tracker-worker;after=1703091894000;before=1703097151000;theme=slate;utc=Europe/Paris.