nfsen 1.3.9 and Profiles with nfdump 1.7.1

[gondimcodes]

Dear Phaag,

I'm using Debian 12 (Bookworm) and packages from the Debian distribution itself:

nfdump-1.7.1-2+deb12u1 nfdump-sflow-1.7.1-2+deb12u1

I installed as usual and configured nfsen 1.3.9. The live profile works correctly but when I try to create other profiles it doesn't work, the flow files are not placed where they should, from what I noticed. I checked and on Debian the package is compiled with --enable-nfprofile support. Is it because it will only work with nfdump 1.7.2? One other thing I noticed was that nfsen was not correctly identifying the version of nfdump and was using the -l parameter to nfcapd instead of the -w. As I don't program in perl I did this here in /data/nfsen/libexec/NfSenRC.pm:

     #my $nfdump_version = $$NfSen::hints{'nfdump'};
     my $nfdump_version = 7;

Can you help me?

[phaag]

Could you please try to install the latest Github nfsen/main branch and lastest Github nfdump/master branch?

[gondimcodes]

Hi Peter, First of all, I would like to thank you for this project. It is very important for us who work with flow to create graphics that help us in network and security engineering. I tried just now using the nfdump version of git 1.7.2 and it still didn't work creating profiles. It's like nfprofile is not working but it is compiled and exists on the system. I compiled with these options: ./configure --enable-nfpcapd --enable-nfprofile --enable-nftrack --enable-nsel --enable-sflow --enable-shared=no

Even so, nfsen fails to create the profiles.

[phaag]

If nfsen is still using -l, then you are not using the NfSen of GitHub/main branch. Please re-install NfSen after upgrading nfdump.

[gondimcodes]

Hi Peter, Now it's working. I had used the nfdump zip file on github and it had not worked. It seems that doing as you recommended worked very well. Thank you very much.

Could you answer one more question for me? When I try to use Netflow Processing using List Flows with custom output:

%ts %pr %flg %sap %dap %pkt %byt %bps %pps %bpp %fl

nfsen returns this error in red: ERROR: nfsend: Illegal characters in argument list!

[gondimcodes]

Hi Peter,

I opened a bug report in Debian to fix this problem with the version of nfdump made available in Debian Bookworm. Only they need information on which exact commit from git master fixed this issue so they can release an already fixed version of the package to Debian. Could you help us? The bug report was opened here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1042535

[phaag]

The error with custom output will get fixed in the main repo

[bernhardschmidt]

Hi Peter,

I opened a bug report in Debian to fix this problem with the version of nfdump made available in Debian Bookworm. Only they need information on which exact commit from git master fixed this issue so they can release an already fixed version of the package to Debian. Could you help us? The bug report was opened here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1042535

Hi @phaag , Debian maintainer of nfdump here. Marcelo reported this bug with Debian (current stable shipped with 1.7.1) and mentioned that 1.7.2 still showed the error for him, but current master didn't.

I cannot just update to the latest upstream snapshot in a stable release so I would need the commit fixing it, ideally applying to the version currently in stable. Marcelo thinks it could be 18a34c16b6d070323d3d44cb22af48a85ac9b0c5 . Does backporting this patch alone sound reasonable to you?

[phaag]

Would it work to wait for September? Then the next release is planned anyway.

[bernhardschmidt]

Yes and no. I can easily package git snapshots and upload them to unstable. However, right now my big concern is the recently released Debian 12 (aka Bookworm) stable release. It includes nfdump 1.7.1 and Debian policy is to keep the upstream version and only cherry-pick targetted bug fixes. It probably was a mistake to rush 1.7 into stable, but the error has been done and cannot be reverted.

I tried backporting 18a34c16b6d070323d3d44cb22af48a85ac9b0c5 and it actually got worse on my own nfsen installation. Quoting from the bugreport

With my installation (still on Bullseye) the official backport of 1.7.1 somewhat works. A few random channels in a profile sometimes show 0 and throw errors like

Uncompress_Block_LZO() error decompression failed in nffile.c line 342: LZO error: -6 Unable to read appendix block of file: /opt/nfsen/profiles-data/ECIX/ECIX-BER-in/2023/08/18/nfcapd.202308181520 Uncompress_Block_LZO() error decompression failed in nffile.c line 342: LZO error: -6 Uncompress_Block_LZO() error decompression failed in nffile.c line 342: LZO error: -6 Unable to read appendix block of file: /opt/nfsen/profiles-data/ECIX/ECIX-MUC-in/2023/08/18/nfcapd.202308181520 Uncompress_Block_LZO() error decompression failed in nffile.c line 342: LZO error: -6 Uncompress_Block_LZO() error decompression failed in nffile.c line 342: LZO error: -6 Uncompress_Block_LZO() error decompression failed in nffile.c line 342: LZO error: -6 ptr error - elementHeader > eor

when running a query against it, but it mostly works.

The patch you mentioned can be applied with just a single manually fixed reject, it builds cleanly and the testsuite also works. But with this patch it actually is worse, no data is ever created by nfprofile. No error in the logs.

Plain 1.7.2 does not work either, same issue.

On 1.7.2 the patch you mentioned does not apply either, it has other rejects. Looking at the changes src/lib/nffile.c had since 1.7.2 has been released I would not be comfortable to do this.

There is way too much movement in nfdump development right now to be able to cherry-pick fixes directly from HEAD, so I'll probably wait for 1.7.3, upload it to unstable and then backport it to bookworm-backports. This way users getting hit by the error are able to explicitly install 1.7.3 from backports.

[phaag]

nfdump-1.7.3 released.

[jult]

Doing the make, on a clean deb bookworm server, I get this at the end:

ut/liboutput.a ../maxmind/libmaxmind.a -lpthread -latomic -lcurl -lbz2 -lzstd -lresolv -pthread
/usr/bin/ld: ../output/liboutput.a(output_csv.o): in function `csv_record':
/root/nf/nfdump/src/output/output_csv.c:125: undefined reference to `FlagsString'
/usr/bin/ld: ../output/liboutput.a(output_fmt.o): in function `String_Flags':
/root/nf/nfdump/src/output/output_fmt.c:1457: undefined reference to `FlagsString'
/usr/bin/ld: ../output/liboutput.a(output_json.o): in function `stringEXgenericFlow':
/root/nf/nfdump/src/output/output_json.c:93: undefined reference to `FlagsString'
/usr/bin/ld: ../output/liboutput.a(output_raw.o): in function `stringsEXflowMisc':
/root/nf/nfdump/src/output/output_raw.c:256: undefined reference to `FlowEndString'
/usr/bin/ld: /root/nf/nfdump/src/output/output_raw.c:256: undefined reference to `biFlowString'
/usr/bin/ld: ../output/liboutput.a(output_raw.o): in function `stringEXgenericFlow':
/root/nf/nfdump/src/output/output_raw.c:115: undefined reference to `FlagsString'
/usr/bin/ld: /root/nf/nfdump/src/output/output_raw.c:123: undefined reference to `FlagsString'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:434: nfdump] Error 1
make[2]: Leaving directory '/root/nf/nfdump/src/nfdump'
make[1]: *** [Makefile:436: all-recursive] Error 1
make[1]: Leaving directory '/root/nf/nfdump'
make: *** [Makefile:368: all] Error 2

likewise, the make install then ends with:

/usr/bin/ld: /root/nf/nfdump/src/output/output_raw.c:123: undefined reference to `FlagsString'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:434: nfdump] Error 1
make[1]: Leaving directory '/root/nf/nfdump/src/nfdump'
make: *** [Makefile:436: install-recursive] Error 1