search

phaag / nfdump

Netflow processing tools
Other
776 stars 203 forks source link

nfdump -M aborting on missing time buckets #479

Closed thezoggy closed 11 months ago

thezoggy commented 11 months ago

Is there a way to have nfdump not abort when time buckets may be missing for a router profile?

went to use the update option, however a recently turned up router on the 12th lacks data for the 8th for this query, so it aborted:

nfdump -M /data/nfsen/profiles-data/live@ -R 2023/10/08/nfcapd.202310080230:2023/10/08/nfcapd.202310080235 -o line -n 20 'proto udp and dst host 82.170.137.1 and src port in [60810]'
stat() error '/data/nfsen/profiles-data/live/a02_<removed>/2023/10/08/nfcapd.202310080230': File not found!

netflow:/data/nfsen/profiles-data/live/a02_<removed>/2023/10> ls
12  13  14  15

Originally posted by @thezoggy in https://github.com/phaag/nfdump/issues/476#issuecomment-1763450263

phaag commented 11 months ago

In your series of input files, -M .. compiles a list of profile directories to be combine with -R ... file list. In order to validate the user input, and to compile a valid file list with potential sub dirs, the first file as well as the last file in the first profile must exist in order to prepare a valid file filter. Otherwise the file list may be invalid and could invalid results are returned. This means the files in the first profile of -M needs to exist in order to build a valid file filter.

thezoggy commented 11 months ago

Iirc When I do the expanded profile method (not the @ expansion you added). The query works, just get the stat warning for the missing file which is fine as expected and doesn't abort.

phaag commented 11 months ago

The order of the expansion is system dependant - it could be alphabetically but need not to be. If by accident the first profile is the one, missing the files, you end up the same.