activate pcp archive compression

redhat-developer / osd-monitor-poc

8 stars 20 forks source link

activate pcp archive compression #15

Open fche opened 6 years ago

fche commented 6 years ago

This should in theory reduce our PV disk space requirements by a factor greater than 10, though in exchange for more temporary RAM. performancecopilot/pcp#422 https://github.com/performancecopilot/pcp/issues/386

fche commented 6 years ago

This also requires pmwebd side changes to process compressed archives properly (looking for .xz file extensions, filename generation during ac_refresh timestamping).

fche commented 6 years ago

also previously needed:

testing to confirm that memory-rlimit forced fallback for decompression works in libpcp; i.e., try to keep going with /tmp-based decompression if -ENOMEM
even better, permit a decompression-buffer memory budget to be set ahead of time (env var?), so we throttle on-the-fly stuff before hitting kernel-level ENOMEM

goodwinos commented 5 years ago

@fche is this still a blocker for OSIO? In any case - before embarking on this feature, you mentioned you were going to manually compress a largish archive collection and see how well pmwebd works with transparent decompression (for both memory and CPU resource usage)

goodwinos commented 5 years ago

@fche what was are your conclusions after testing? IIRC we discussed this a while back and tentatively concluded the overheads were too high for pmwebd, especially on startup. Is that correct?

fche commented 5 years ago

The overheads -are- high. But since the need still exists to ease disk usage and/or to allow longer term retention, the capability is still needed.

fche commented 5 years ago

The overheads -are- high. But since the need still exists to ease disk usage and/or to allow longer term retention, the capability is still needed.

By the way, one way to reduce that overhead somewhat is to keep archives open longer. https://bugzilla.redhat.com/show_bug.cgi?id=1619708#c9