nfsen - setting compression value

[thezoggy]

moved original nfdump question over to nfdump - updated this one to focus on just nfsen

Recently I upgraded to nfdump and saw nfcapd no longer being stored in lz4 even thought I saw it still being set to use it. I wanted to test if updating to another compression such as zstd fixed it and ran into an issue.

looking at nfsen.conf I see it was using the legacy value:

$ZIPprofiles     = 1;

looking at nfsen status, I do see it is doing -z=lz4 to nfcapd...

> sudo systemctl status nfsen
● nfsen.service - NfSen Service
     Loaded: loaded (/etc/systemd/system/nfsen.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-05-17 23:36:27 UTC; 11s ago
    Process: 2193948 ExecStart=/data/nfsen/bin/nfsen start (code=exited, status=0/SUCCESS)
      Tasks: 945 (limit: 38372)
     Memory: 9.0G
     CGroup: /system.slice/nfsen.service
             ├─2193959 /usr/local/bin/nfcapd -D -p 40054 -u www-data -g www-data -B 200000 -S 1 -P /data/nfsen/var/run/p40054.pid -z=lz4 -t 300 -I router1 -w /data/nfsen/profiles-data/live/router1 -s 5000
             ├─2193966 /usr/local/bin/nfcapd -D -p 40036 -u www-data -g www-data -B 200000 -S 1 -P /data/nfsen/var/run/p40036.pid -z=lz4 -t 300 -I router2 -w /data/nfsen/profiles-data/live/router2 -s 5000

Looking at: https://github.com/phaag/nfsen/blob/main/etc/nfsen-dist.conf#L126

I wanted to try using zstd so I updated:

- $ZIPprofiles   = 1;
+ $ZIPprofiles   = "-z=zstd:3";

This did not change what is being passed to nfcapd...

digging into nfconf.. https://github.com/phaag/nfsen/blob/main/libexec/NfConf.pm#L152

I see its "$ZIPcollected" is what needs to be changed... nfsen-dist.conf never got updated to note it like ZIPprofiles did.

# Compress flows while collecting 0 or 1
$ZIPcollected    = 1;

Waiting for some stuff to finish, but will updated nfsen conf to test and confirm:

- $ZIPcollected  = 1;
+ $ZIPcollected  = "-z=zstd:3";

[phaag]

I am sorry for the trouble, but between the two master version, there is no change for backend or expire behavior. It looks to me, that that all was a chaining of unlucky events! The file nfsen-dist.conf always reflected all available variables. The change from …=1; to …= “-z= xx” was introduced mainly because of users wanted to have different compression available in NfSen. In oder to be backward compatible, 1 was mapped to lz4. The NfSen main branch should be up to date to reflect all that. The way you collect and process the flows is controlled be NfSen and its config.

As of lz4: It was a user demand to explicitly use the system installed lz4 library and not the code, which comes with nfdump. However, as I still consider lz4 as one of the optimal compression, which should be available in nfdump, the lz4 code still is shipped with nfdump. The configure command checks for the system installed lz4 library and if it is not available it builds the shipped lz4 code - the embedded version. Therefore, its a matter of taste, which one you prefer.

Finally please note, if you change the compression of the files, you need the rebuild the profile in order to reflect the correct new size of the profile.

[thezoggy]

Updated title and original notes since, also moved original problem with compression not working to nfdump repo for that. Keeping this for only the nfsen conf one

[phaag]

There is no change, which stops nfcapd from compressing with lz4.