nfsen-ng: 400 - Bad Request. rrd_xport failed.

[hans-mayer]

Dear All,

I am not sure if I am right here. I didn't find a mailing for "nfsen-ng"

My issue I don't get any data on the GUI.

I followed the instructions at https://github.com/mbolli/nfsen-ng to install on a fresh installed "Ubuntu 18.04.1 LTS" "nfdump" wasn't found as packaged, so I compiled from source.

This is my file nfsen-ng/backend/settings/settings.php

<?php
/**
 * config file for nfsen-ng
 *
 * remarks:
 * * database name = datasource class name (case-sensitive)
 * * log priority should be one of the predefined core constants prefixed with LOG_
 */

$nfsen_config = array(
    'general' => array(
        'ports' => array(
            80, 22, 53,
        ),
        'sources' => array(
            'cs-sl0-s6506',
        ),
        'db' => 'RRD',
    ),
    'frontend' => array(
        'reload_interval' => 60,
    ),
    'nfdump' => array(
        'binary' => '/usr/bin/nfdump',
        'profiles-data' => '/var/nfdump/profiles-data',
        'profile' => 'live',
        'max-processes' => 2, // maximum number of concurrently running nfdump processes
    ),
    'db' => array(
        'Akumuli' => array(
            //'host' => 'localhost',
            //'port' => 8282,
        ),
        'RRD' => array()
    ),
    'log' => array(
        'priority' => LOG_DEBUG, // LOG_INFO, LOG_DEBUG is very talkative!
    )
);

I run

nfcapd -w -D -S 2 -B 1024000 -l /var/nfdump/profiles-data/live/cs-sl0-s6506 -p 10001

which writes sucessfully data in subdiretories like this

/var/nfdump/profiles-data/live/cs-sl0-s6506/2018/11/13/16/nfcapd.201811131620

But when I open my web-browser I get an error message:

Got 400 - Bad Request. rrd_xport failed. opening '/var/www/html/nfsen-ng/backend/datasources/data/cs-sl0-s6506.rrd': No such file or directory

and syslog shows at the same time:

Nov 13 16:52:52 hpg911 apache2: nfsen-ng: Was not able to find /var/www/html/nfsen-ng/backend/datasources/data/cs-sl0-s6506.rrd Nov 13 16:52:52 hpg911 apache2: nfsen-ng: 400 - Bad Request. rrd_xport failed. opening '/var/www/html/nfsen-ng/backend/datasources/data/cs-sl0-s6506.rrd': No such file or directory

I do not know where to configure a process to convert the pcap data to rrd files. I gave full permissions for /var/www/html/nfsen-ng/backend/datasources/data/ so it shouldn't be a permissions issue.

RRDtool 1.7.0 is installed,

apachectl -M shows php7_module is installed, but nothing about rrd, probably OK.

"pecl list" tells me

rrd 2.0.1 stable

I started manually "./cli.php start"

./cli.php status Running: 14178

Any help is welcome.

// Hans

[mbolli]

Hi Hans

First, you're correct in posting the issue/question here. It is the issues queue after all. Second, there is really a package for nfdump in Ubuntu, but you have to enable the Universe repository apparently. I am going to add this to the installation instructions. Third, the process to convert the pcap data to rrd files is done by nfdump and nfsen-ng. But it looks like it can't write to the destination file (cs-sl0-s6506.rrd). In the write() function in the RRD class it tries to create the file if it doesn't exist, although it doesn't emit an error if that failed. Can you pull the master where this is fixed? Then we know where exactly the problem lies.

Regards Michael

[hans-mayer]

Dear Michael,

I followed your instructions to install the universe repository. I could install this nfdump package. interesting I have rdd file only from 10:15 h.

root@hpg911:/var/www/html# ls -l /var/www/html/nfsen-ng/backend/datasources/data/
total 37324
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 22.rrd
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 53.rrd
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 80.rrd
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 cs-sl0-s6506_22.rrd
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 cs-sl0-s6506_53.rrd
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 cs-sl0-s6506_80.rrd
-rw-r--r-- 1 www-data www-data 5457448 Nov 14 10:15 cs-sl0-s6506.rrd

The web page doesn't show any error any more. But of course there are no data.

In nfsen-ng.log there are only warnings, but a lot of them

PHP Notice:  Undefined offset: 1 in /var/www/html/nfsen-ng/backend/common/import.php on line 107
Notice: Undefined offset: 1 in /var/www/html/nfsen-ng/backend/common/import.php on line 107

I enabled "debug" for syslog, but there are no error messages.

Is there a way to run "nfdump" with all its options and arguments once from the command line and see if there are any errors ? and to verify what are the results ?

I forgot to mention: I am sending data from a Cisco 6506 with

ip flow-export version 9
ip flow-aggregation cache protocol-port
ip flow-top-talkers

and the necessary config in various interface declarations for IPv4 and IPv6

// Hans

[hans-mayer]

Dear Michael,

I forgot to say, of course I did a "git pull". But this didn't show the issues.

// Hans

[mbolli]

PHP Notice:  Undefined offset: 1 in /var/www/html/nfsen-ng/backend/common/import.php on line 107
Notice: Undefined offset: 1 in /var/www/html/nfsen-ng/backend/common/import.php on line 107

Can you post a listing of your nfcapd.* files in your profiles-data directory? In the master branch I added a log output if the files have the wrong file name syntax, if you'd like to pull once more.

Is there a way to run "nfdump" with all its options and arguments once from the command line and see if there are any errors ? and to verify what are the results ?

Yes of course, nfdump is a normal command line tool, for instance:

$ nfdump -M cs-sl0-s6506 -r 2018/11/15/nfcapd.201811151000 -s dstport:p "dst port=80" -o csv
ts,te,td,pr,val,fl,flP,ipkt,ipktP,ibyt,ibytP,ipps,ipbs,ibpp
2017-04-30 14:59:38,2017-04-30 15:04:31,293.000,TCP,80,90,98.9,100,99.0,5654,99.2,0,154,56
2017-04-30 15:01:47,2017-04-30 15:01:47,0.000,UDP,80,1,1.1,1,1.0,46,0.8,0,0,46

Summary
flows,bytes,packets,avg_bps,avg_pps,avg_bpp
91,5700,101,155,0,56

[hans-mayer]

Dear Michael,

Thanks for coming back.

Can you post a listing of your nfcapd.* files in your profiles-data directory?

www-data@hpg911:~/html/nfsen-ng$ ls -la /var/nfdump/profiles-data/live/cs-sl0-s6506/2018/11/16/00
total 1804
drwxrwxr-x 2 www-data www-data   4096 Nov 16 01:00 .
drwxrwxr-x 6 www-data www-data   4096 Nov 16 03:05 ..
-rw-r--r-- 1 www-data www-data  66668 Nov 16 00:05 nfcapd.201811160000
-rw-r--r-- 1 www-data www-data 113560 Nov 16 00:10 nfcapd.201811160005
-rw-r--r-- 1 www-data www-data 120724 Nov 16 00:15 nfcapd.201811160010
-rw-r--r-- 1 www-data www-data  82708 Nov 16 00:20 nfcapd.201811160015
-rw-r--r-- 1 www-data www-data  76640 Nov 16 00:25 nfcapd.201811160020
-rw-r--r-- 1 www-data www-data  85208 Nov 16 00:30 nfcapd.201811160025
-rw-r--r-- 1 www-data www-data 855992 Nov 16 00:35 nfcapd.201811160030
-rw-r--r-- 1 www-data www-data 105204 Nov 16 00:40 nfcapd.201811160035
-rw-r--r-- 1 www-data www-data  78596 Nov 16 00:45 nfcapd.201811160040

In the master branch I added a log output if the files

There are no errors reported in nfsen-ng.log and a lot of processing statements. This seems to be OK. log priority I have LOG_ERR. And this is the version I have

www-data@hpg911:~/html/nfsen-ng$ git log -1
commit 0935f353c7bba85d4b7eefd1b4f66e4f95726edb (HEAD -> master, origin/master, origin/HEAD)
Author: Michael Bolli <michael@moxi.ch>
Date:   Thu Nov 15 08:51:00 2018 +0100

    import: log bad nfcapd file formats

Yes of course, nfdump is a normal command line tool, for instance:

But this is obviously not in the same way as nfsen-ng is calling it. Output should be a rrd file and not a csv, isn't it ?

As I mentioned I compiled the source for nfdump and later on I installed the package. The self compiled has Version: 1.6.17 , the package is Version: NSEL-NEL1.6.16 I tried with both your example, but none of them is working.

www-data@hpg911:~/html/nfsen-ng$ cd /var/nfdump/profiles-data/live/

the package version

www-data@hpg911:/var/nfdump/profiles-data/live$ ls -l cs-sl0-s6506/2018/11/17/20/nfcapd.201811172020                                
-rw-r--r-- 1 www-data www-data 13428 Nov 17 20:25 cs-sl0-s6506/2018/11/17/20/nfcapd.201811172020

www-data@hpg911:/var/nfdump/profiles-data/live$ /usr/bin/nfdump -M cs-sl0-s6506 -r 2018/11/17/20/nfcapd.201811172020 -s dstport:p "dst port=80" -o csv -s dstport:p "dst port=80" -o csv ts,te,td,pr,val,fl,flP,ipkt,ipktP,ibyt,ibytP,ipps,ipbs,ibpp

usage /usr/bin/nfdump [options] ["filter"]
-h              this text you see right here
-V              Print version and exit.
-a              Aggregate netflow data.
-A <expr>[/net] How to aggregate: ',' sep list of tags see nfdump(1)
                or subnet aggregation: srcip4/24, srcip6/64.
-b              Aggregate netflow records as bidirectional flows.
-B              Aggregate netflow records as bidirectional flows - Guess direction.
-r <file>       read input from file
-w <file>       write output to file
-f              read netflow filter from file
-n              Define number of top N for stat or sorted output.
-c              Limit number of records to read from source(es)
-D <dns>        Use nameserver <dns> for host lookup.
-N              Print plain numbers
-s <expr>[/<order>]     Generate statistics for <expr> any valid record element.
                and ordered by <order>: packets, bytes, flows, bps pps and bpp.
-q              Quiet: Do not print the header and bottom stat lines.
-H Add xstat histogram data to flow file.(default 'no')
-i <ident>      Change Ident to <ident> in file given by -r.
-J <num>        Modify file compression: 0: uncompressed - 1: LZO compressed - 2: BZ2 compressed.
-z              lzo compress flows in output file. Used in combination with -w.
-j              bz2 compress flows in output file. Used in combination with -w.
-l <expr>       Set limit on packets for line and packed output format.
                key: 32 character string or 64 digit hex string starting with 0x.
-L <expr>       Set limit on bytes for line and packed output format.
-I              Print netflow summary statistics info from file, specified by -r.
-M <expr>       Read input from multiple directories.
                /dir/dir1:dir2:dir3 Read the same files from '/dir/dir1' '/dir/dir2' and '/dir/dir3'.
                requests either -r filename or -R firstfile:lastfile without pathnames
-m              deprecated
-O <order> Sort order for aggregated flows - tstart, tend, flows, packets bps pps bbp etc.
-R <expr>       Read input from sequence of files.
                /any/dir  Read all files in that directory.
                /dir/file Read all files beginning with 'file'.
                /dir/file1:file2: Read all files from 'file1' to file2.
-o <mode>       Use <mode> to print out netflow records:
                 raw      Raw record dump.
                 line     Standard output line format.
                 long     Standard output line format with additional fields.
                 extended Even more information.
                 csv      ',' separated, machine parseable output format.
                 pipe     '|' separated legacy machine parseable output format.
                        mode may be extended by '6' for full IPv6 listing. e.g.long6, extended6.
-E <file>       Print exporter ans sampling info for collected flows.
-v <file>       verify netflow data file. Print version and blocks.
-x <file>       verify extension records in netflow data file.
-X              Dump Filtertable and exit (debug option).
-Z              Check filter syntax and exit.
-t <time>       time window for filtering packets
                yyyy/MM/dd.hh:mm:ss[-yyyy/MM/dd.hh:mm:ss]

self compiled

www-data@hpg911:/var/nfdump/profiles-data/live$ /usr/bin/nfdump_1617 -M cs-sl0-s6506 -r 2018/11/17/20/nfcapd.201811172020 -s dstport:p "dst port=80" -o csv -s dstport:p "dst port=80" -o csv ts,te,td,pr,val,fl,flP,ipkt,ipktP,ibyt,ibytP,ipps,ipbs,ibpp
Invalid sampling rate: dstport:p

Is there a way where I can send you one of my nfcapd-files ? ( maybe not public available )

Do you have any ideas what I can try ?

Schöne Grüße aus Wien // Hans

[mbolli]

Oh I see what the problem is: the path syntax of your capture directory is wrong: You have YYYY/MM/DD/HH, but correct is YYYY/MM/DD. Can you try to fix that and see if it changes anything?

[hans-mayer]

Dear Michael,

Thanks for coming back.

Unfortunately it doesn't work. I have now a nfcap daemon running

nfcapd -w -D -S 1 -B 1024000 -l /var/nfdump/profiles-data/live/cs-sl0-s6506 -p 10001

which generates files like this

/var/nfdump/profiles-data/live/cs-sl0-s6506/2018/11/22/nfcapd.201811221035
/var/nfdump/profiles-data/live/cs-sl0-s6506/2018/11/22/nfcapd.201811221025
/var/nfdump/profiles-data/live/cs-sl0-s6506/2018/11/22/nfcapd.201811221030

In my setting I have

        'profiles-data' => '/var/nfdump/profiles-data',
        'profile' => 'live',

But directory /var/www/html/nfsen-ng/backend/datasources/data/ stays empty
Where is defined that /var/www/html/nfsen-ng/backend/datasources/data should be the destination ? And the web browser says:

Got 400 - Bad Request. rrd_xport failed. opening '/var/www/html/nfsen-ng/backend/datasources/data/cs-sl0-s6506.rrd': No such file or directory

do I miss a program 'xxd_xport' ? I would like to know how can I run a command line which converts data from /var/nfdump/profiles-data/live/cs-sl0-s6506/* to the necassary rrd files in /var/www/html/nfsen-ng/backend/datasources/data ? Is there a way to do it manually from the command line ? I can't see an option for nfdump generating rrd files.

// Hans

[mbolli]

I have now a nfcap daemon running which generates files like this

Looks good.

But directory /var/www/html/nfsen-ng/backend/datasources/data/ stays empty

Truly weird.

Where is defined that /var/www/html/nfsen-ng/backend/datasources/data should be the destination ?

That's hardcoded for the RRD data source.

And the web browser says:

Got 400 - Bad Request. rrd_xport failed. opening '/var/www/html/nfsen-ng/backend/datasources/data/cs-sl0-s6506.rrd': No such file or directory

It somehow still looks like a permissions problem of creating the database files. Can you set log priority to LOG_DEBUG again, and start the daemon? Maybe the log file will show something...

do I miss a program 'xxd_xport' ?

No, or the error above would not appear.

I would like to know how can I run a command line which converts data from /var/nfdump/profiles-data/live/cs-sl0-s6506/* to the necassary rrd files in /var/www/html/nfsen-ng/backend/datasources/data ? Is there a way to do it manually from the command line ?

No there isn't. Parsing the nfdump-returned data as well as writing it to the RRD databases is done by nfsen-ng.

Grüsse aus Biel/CH Michael

[Akshaykkapoor]

I am facing the same issue , do we have a solution ?

[thebadgateway]

I am facing the same issue , do we have a solution ?

how did you solve the problem?

[hans-mayer]

Dear All,

I got a lot of support from Michael but couldn't solve this issue running nfsen-ng successfully on a fresh installed "Ubuntu 18". With lack of time I postponed this project. So I have no solution currently. But I would be happy to hear about progress from others. Hopefully next year I have more time to continue on this.

Kind regards Hans

--

[mbolli]

Sorry guys I as well did not have the time to investigate further. Until then, happy holidays!

[hans-mayer]

Dear All,

I have some news: it's working for me. Not sure if this is necessary, but I installed and run "Akumili". The database files didn't change since I started it. But at least there is a reference in settings/settings.php for 'Akumuli'

Definetelly there was an issue with the used nfdump version: Version: 1.6.17 does NOT work. Version: NSEL-NEL1.6.16 is working.

Kind regards Hans

[hans-mayer]

Dear All,

I close my ticket as there is a solution for me.

Kind regards Hans

[FontouraAbreu]

For anyone who is facing this same issue:

For me, what did the trick was

1. Correcting the /etc/nfdump/default.conf option var

options='-z -S 1 -T all -l /var/nfdump/profiles-data/live/<source> -p <port>'

Where (nfcapd doc.):

 -S <num>
          Allows to specify an additional directory sub hierarchy to store the  data  files.  The
          default is 0, no sub hierarchy, which means the files go directly in the base directory
          (-l). The base directory (-l) is concatenated with the specified sub  hierarchy  format
          to form the final data directory.  The following hierarchies are defined:
            0 default     no hierarchy levels
            1 %Y/%m/%d    year/month/day
            2 %Y/%m/%d/%H year/month/day/hour
            3 %Y/%W/%u    year/week_of_year/day_of_week
            4 %Y/%W/%u/%H year/week_of_year/day_of_week/hour
            5 %Y/%j       year/day-of-year
            6 %Y/%j/%H    year/day-of-year/hour
            7 %Y-%m-%d    year-month-day
            8 %Y-%m-%d/%H year-month-day/hour

2. Correcting the 'profiles-data' path variable in nfsen-ng/backend/settings/settings.php to '/var/nfdump/profiles-data/live/'

At the current time, the default value is '/var/nfdump/profiles-data/'

3. Restarting nfsen-ng

I created a systemd service but if you haven't, it should be simply nfsen-ng/backend/cli.php start | stop

After these, the nfsen-ng/backend/datasources/data/ dir was simply created and filled with the .rrd files. Accessing the nfsen-ng web-page now shows the graphs

[davidshirk80]

@FontouraAbreu - That works and I Have no errors now - but no data displayed either :-/ rrd files are made and the ncapd is running right

nfcapd -z -S 1 -T all -l /var/nfdump/profiles-data/live/source -p 2055

root@un:system:~# ls -la /var/www/html/nfsen-ng/backend/datasources/data total 21336 drwxrwxrwx 2 root root 4096 Feb 7 22:45 . drwxr-xr-x 3 www-data www-data 4096 Feb 7 22:13 .. -rw-r--r-- 1 root root 5457448 Feb 7 23:10 source_22.rrd -rw-r--r-- 1 root root 5457448 Feb 7 23:10 source_53.rrd -rw-r--r-- 1 root root 5457448 Feb 7 23:10 source_80.rrd -rw-r--r-- 1 root root 5457448 Feb 7 23:10 source rrd

Bound to IPv4 host/IP: any, Port: 2055 Startup. Init IPFIX: Max number of IPFIX tags: 72 Process_v9: New exporter: SysID: 1, Domain: 0, IP: source IP

Process_v9: [0] Add template 257 Process_v9: [0] Add template 256 Ident: 'none' Flows: 28531, Packets: 1186891, Bytes: 281461716, Sequence Errors: 0, Bad Packets: 0 Total ignored packets: 0

root@un:system:~# ls -la /var/nfdump/profiles-data/live/source total 696 drwxrwxrwx 3 root root 4096 Feb 7 23:15 . drwxrwxrwx 3 root root 4096 Feb 7 23:11 .. drwxr-xr-x 3 root root 4096 Feb 7 23:15 2024 -rw-r--r-- 1 root root 696637 Feb 7 23:19 nfcapd.current.15070

And nothing is showing on the webpage. No errors, but no data either. Ideas?

[davidshirk80]

Here is my settings config: /// $nfsen_config = array( 'general' => array( 'ports' => array( 80, 22, 53, ), 'sources' => array( 'source', ), 'db' => 'RRD', 'processor' => 'NfDump', ), 'frontend' => array( 'reload_interval' => 60, 'defaults' => array( 'view' => 'graphs', // graphs, flows, statistics 'graphs' => array( 'display' => 'sources', // sources, protocols, ports 'datatype' => 'flows', // flows, packets, bytes 'protocols' => array('any'), // any, tcp, udp, icmp, others (multiple possible if display=protocols) ), 'flows' => array( 'limit' => 50, ), 'statistics' => array( 'order_by' => 'bytes', ), ), ), 'nfdump' => array( 'binary' => '/usr/bin/nfdump', 'profiles-data' => '/var/nfdump/profiles-data/live', 'profile' => 'live', 'max-processes' => 1, // maximum number of concurrently running nfdump processes ), 'db' => array( 'Akumuli' => array( //'host' => 'localhost', //'port' => 8282, ), 'RRD' => array() ), 'log' => array( 'priority' => LOG_INFO, // LOG_DEBUG is very talkative! ) ); ///

[davidshirk80]

Also, how did you create a start for this service?

[FontouraAbreu]

@davidshirk80 -

That works and I Have no errors now - but no data displayed either :-/ rrd files are made and the ncapd is running right And nothing is showing on the webpage. No errors, but no data either. Ideas?

I've actually run into the same problem and I'm still trying to figure it out about what maybe causing it.

At first my source was sending only sflow, not netflow, so I just changed /lib/systemd/system/nfdump@.service 'ExecStart' and 'PIDFile' variable from:

[Service]
Type=forking
EnvironmentFile=/etc/nfdump/%I.conf
ExecStart=/usr/bin/nfcapd -w -D -P -S 1 -B 200000 /run/nfcapd.%I.pid $options
PIDFile=/run/nfcapd.%I.pid

to:

[Service]
Type=forking
EnvironmentFile=/etc/nfdump/%I.conf
ExecStart=/usr/bin/sfcapd -w /var/nfdump/profiles-data/live/source -D -S 1 -B 200000 -P /run/sfcapd.%I.pid $options
PIDFile=/run/sfcapd.%I.pid

And then, I was receiving all sflow sent from the source but nothing is showing on my webpage either.

Also, how did you create a start for this service?

I've created /lib/systemd/system/nfsen-ng.service:

[Unit]
Description=nfsen-ng
After=network-online.target

[Service]
Type=simple
RemainAfterExit=yes
ExecStart=su - www-data --shell=/bin/bash -c '/var/www/html/nfsen-ng/backend/cli.php start'
ExecStop=su - www-data --shell=/bin/bash -c '/var/www/html/nfsen-ng/backend/cli.php stop'

[Install]
WantedBy=multi-user.target

And then, I just needed to reload the systemctl daemon and enable the service:

systemctl daemon-reload
systemctl enable nfsen-ng
systemctl start nfsen-ng
systemctl status nfsen-ng

A colleague of mine suggested to compile nfdump from source and try using it. That's what I'll be trying. I hope it helps.

[FontouraAbreu]

I'd like to know if @mbolli prefers that we created another issue or keep commenting in this one? Since we got here from probably the same Got 400 - Bad Request. rrd_xport failed. opening '/var/www/html/nfsen-ng/backend/datasources/data/source.rrd': No such file or directory error.

[davidshirk80]

Not sure - I am wondering if @mbolli runs into the same issue on a fresh Ubuntu 22.04 install :-/ I had to create the var/nfdump folder as well as the subsequent folders and assign permissions. After freshly installing, I noted that nfdump did not create a folder in /var/nfdump - so it appears that the version of nfdump in 22.04 is functioning and working entirely differently from where this program expects it to be. Going to to try to compile from source as well and check it it out.

Also - thank you for the adding service enable directions. I am a network engineer not true coder, so this is a huge learning curve for me!

[davidshirk80]

Also as to nfdump - there is the one that comes in in multiverse/universe - and then there is the one by phaag on github. I am going to rebuild the box and try the one from phaag as the one from universe/multiverse does not seem to play well here. phaag nfdump only seems to have install instructions for 18.04 - so I do not want to put that on a 22.04 box

[davidshirk80]

I NOTICED THIS IN THE SETUP DIRECTIONS "Expected NFDUMP info: PROFILES_DATA_PATH/PROFILE/SOURCE/YYYY/MM/DD/nfcapd.YYYYMMDDHHII, e.g. /var/nfdump/profilesdata/live/source1/2018/12/01/nfcapd.201812010225" - the being different than the -. Going to reprogram using the underscore and see what happens.

[mbolli]

@FontouraAbreu @davidshirk80 Is this still related to the nfdump version? In https://github.com/mbolli/nfsen-ng/issues/17#issuecomment-449507391 @hans-mayer states that it doesn't work with nfdump v1.6.17. What nfdump versions are you guys running?

Here's how I compiled the most recent nfdump version on a recent Debian:

wget https://github.com/phaag/nfdump/archive/refs/tags/v1.7.3.zip
apt install flex libbz2-dev yacc
unzip v1.7.3.zip
cd nfdump-1.7.3/
./autogen.sh
./configure
make
make install
ldconfig
nfdump -V

I'd like to know if @mbolli prefers that we created another issue or keep commenting in this one? Since we got here from probably the same Got 400 - Bad Request. rrd_xport failed. opening '/var/www/html/nfsen-ng/backend/datasources/data/source.rrd': No such file or directory error.

Well this issue has been closed a long time, but since we're all here...

I NOTICED THIS IN THE SETUP DIRECTIONS "Expected NFDUMP info: PROFILES_DATA_PATH/PROFILE/SOURCE/YYYY/MM/DD/nfcapd.YYYYMMDDHHII, e.g. /var/nfdump/profilesdata/live/source1/2018/12/01/nfcapd.201812010225" - the being different than the -. Going to reprogram using the underscore and see what happens.

That's not going to help, PROFILES_DATA_PATH is just a variable for where the data lives on your system

[davidshirk80]

I saw that - but was not sure if was how nfdump was being read or what. What directory did you place nfdump in?

[davidshirk80]

/var it looks like

[FontouraAbreu]

What nfdump versions are you guys running?

root@nfsen-ng:/var/www/html/nfsen-ng/backend# nfdump -V
nfdump: Version: NSEL-NEL1.7.1

I'm using a debian12 machine to set it up:

---

Now, Here is a funny story: Yesterday, i ran into a timezone problem at the VM I'm using to setup nfsen-ng and, by accident, I tried apt purge tzdata after the problem was solved.

The command also purged some php dependencies which broke the nfsen-ng graphs and filter. I did re-install the purged packages and dependencies but didn't manage to get the nfsen-ng graphs and nfdump filters to work . Today, arriving at office and eager to try again from a fresh VM, compiling nfdump from source and documenting every step I opened my local nfsen-ng WebPage and everything is working fine, including the graphs containing the data collected from my source.

Given this computer magic, I investigated for a while an came to some suspicions:

• graphs and filter have a delay when checking new nfcapd files In my source data directory I have these files:

root@nfsen-ng:/var/www/html/nfsen-ng/backend# ls -lt /var/nfdump/profiles-data/live/source/2024/02/09/ | head
total 25164
-rw-r--r-- 1 root root 473619 fev  9 10:20 nfcapd.202402091015
-rw-r--r-- 1 root root 485526 fev  9 10:15 nfcapd.202402091010
-rw-r--r-- 1 root root 479699 fev  9 10:10 nfcapd.202402091005
-rw-r--r-- 1 root root 485602 fev  9 10:05 nfcapd.202402091000
-rw-r--r-- 1 root root 443575 fev  9 10:00 nfcapd.202402090955
-rw-r--r-- 1 root root 470694 fev  9 09:55 nfcapd.202402090950
-rw-r--r-- 1 root root 516209 fev  9 09:50 nfcapd.202402090945
-rw-r--r-- 1 root root 481809 fev  9 09:45 nfcapd.202402090940
-rw-r--r-- 1 root root 465303 fev  9 09:40 nfcapd.202402090935

So, the most recent is from 2024/02/09 at 10h15. My flow graphs only show until 2024/02/09 at 7h15

I still suspect this is a timezone error since my timezone is America/Sao_Paulo which is UTC-3 and the difference between collected data and shown data is exactly 3h

Still, its weird that only after 2 days of collected data the graphs and filters showed me something.

• Application dependencies and configuration are bad documented I did have a hard time understanding what needs to be done in order to install and configure nfsen-ng.

I do am to blame for most of it since it was my first time working with netflow, sflow and nfdump, but still, the documentation doesn't cover theses topics very well and can cause some confusion to someone starting to learn these or coming from a nfsen installation and configuration background.

Having said that, I'd like to know if @mbolli is okay with me writing an alternative installation and configuration documentation based on my experience and difficulties? I'm also planning on dockerizing this nfsen-ng installation as this better suits my context, so I'd also like to add a docker alternative to it.

---

Are there other configuration files similar to /etc/nfsen.conf from the original nfsen project, where I can set some other variables like: graph colour based on ident, default period to display, graph scale, series display and things like that?

[mbolli]

Great that it's working now.

I still suspect this is a timezone error since my timezone is America/Sao_Paulo which is UTC-3 and the difference between collected data and shown data is exactly 3h

Might be connected: https://github.com/mbolli/nfsen-ng/issues/68

Still, its weird that only after 2 days of collected data the graphs and filters showed me something.

True

Having said that, I'd like to know if @mbolli is okay with me writing an alternative installation and configuration documentation based on my experience and difficulties? I'm also planning on dockerizing this nfsen-ng installation as this better suits my context, so I'd also like to add a docker alternative to it.

We are always open for PRs, might be something to include in the upcoming v0.3 release https://github.com/mbolli/nfsen-ng/pull/90

Are there other configuration files similar to /etc/nfsen.conf from the original nfsen project, where I can set some other variables like: graph colour based on ident, default period to display, graph scale, series display and things like that?

No, but you can open a new issue with what configuration options you'd like.

[FontouraAbreu]

Might be connected: https://github.com/mbolli/nfsen-ng/issues/68

Understood the reason but didn't find where I should change in the nfsen-ng/frontend/js/nfsen-ng.js file

[davidshirk80]

Also - installing nfdump from the distro release is 1.6.x and as pointed out will not work - so setup on 22.04 should specify nfdump instructions specifically. Also, manual folders have to be created for it to work on 22.04 and do not aut0-generate. Not on my Ubuntu vm anyway :-/ Thank you for the help though - and also ensuring proper timezone config in php appears to be key!

[davidshirk80]

Made the mistake of starting over on fresh machine using the same variables that appeared to clear the rrd issue earlier, and am right back to the 400 - here is my current info

root@un-machine:~/nfdump-1.7.3# nfdump -V nfdump: Version: 1.7.3-release Date: Fri 09 Feb 2024 11:35:59 AM CST

nfcapd -w -D -S 1 -B 200240209 -l /var/nfdump/profiles-data/live/source -p 2055

root@un-machine:~# ls -la /var/nfdump/profiles-data/live/source1/2024/02/09 total 11252 drwxr-xr-x 2 www-data www-data 4096 Feb 9 13:05 . drwxr-xr-x 3 www-data www-data 4096 Feb 9 11:44 .. -rw-r--r-- 1 www-data www-data 47426 Feb 9 11:44 nfcapd.202402091140 -rw-r--r-- 1 www-data www-data 666834 Feb 9 12:05 nfcapd.202402091200 -rw-r--r-- 1 root root 902701 Feb 9 12:40 nfcapd.202402091235 -rw-r--r-- 1 root root 892777 Feb 9 12:45 nfcapd.202402091240

settings.php has tried both with and without live to same result 'nfdump' => array( 'binary' => '/usr/bin/nfdump', 'profiles-data' => '/var/nfdump/profiles-data/live', 'profile' => 'live', 'max-processes' => 1, // maximum number of concurrently running nfdump processes Correcting the /etc/nfdump/default.conf option var - this file does not exist since I installed nfdump manually root@un-machine:~# ls -la /var/www/html/nfsen-ng/backend/datasources/data/ total 8 drwxrwxrwx 2 root root 4096 Feb 9 11:51 . drwxr-xr-x 3 www-data www-data 4096 Feb 9 11:51 .. root@un-machine:~# nfcapd -z -S 1 -T all -w /var/nfdump/profiles-data/live/source1 -p 2055 Option -T no longer supported and ignored Bound to IPv4 host/IP: any, Port: 2055 Init v5/v7: Default sampling: 1 Init v9: Max number of v9 tags enabled: 105, default sampling: 1 Init IPFIX: Max number of ipfix tags enabled: 91, default sampling: 1 Startup nfcapd. Process_v9: New v9 exporter: SysID: 1, Domain: 0, IP: source1 ip Ident: 'none' Flows: 5077, Packets: 222530, Bytes: 52901885, Sequence Errors: 0, Bad Packets: 0, Blocks: 0 Ident: 'none' Flows: 42379, Packets: 1889725, Bytes: 441151741, Sequence Errors: 0, Bad Packets: 0, Blocks: 0 Ident: 'none' Flows: 42382, Packets: 1984912, Bytes: 458191622, Sequence Errors: 0, Bad Packets: 0, Blocks: 0 Ident: 'none' Flows: 42524, Packets: 2184275, Bytes: 499847795, Sequence Errors: 2, Bad Packets: 0, Blocks: 0 Ident: 'none' Flows: 42524, Packets: 2064786, Bytes: 477182651, Sequence Errors: 0, Bad Packets: 0, Blocks: 0

[davidshirk80]

I used every single fix listed earlier. Checked my ncap files and they are formatted exactly as they should be,

[mbolli]

@davidshirk80 you have some permission issues. Ensure that www-data user/group can write to /var/www/html/nfsen-ng/backend/datasources/data/

[davidshirk80]

As a test, I gave chmod 777 /var/www/html/nfsen-ng/backend/datasources/data/ a go - did not help :-/

[davidshirk80]

root@un-machine:~# ls -la /var/www/html/nfsen-ng/backend/datasources/data/ total 8 drwxrwxrwx 2 root root 4096 Feb 9 13:19 . drwxrwxrwx 3 www-data www-data 4096 Feb 9 11:51 ..

[davidshirk80]

getting this though :-/ Feb 12 16:36:10 machine systemd[1]: Started nfsen-ng. Feb 12 16:36:10 machine su[16390]: (to www-data) root on none Feb 12 16:36:10 machine su[16390]: pam_unix(su-l:session): session opened for user www-data(uid=33) by (uid=0) Feb 12 16:36:11 machine su[16390]: pam_unix(su-l:session): session closed for user www-data

[davidshirk80]

If you could please humor me - and on a fresh 22.04 Ubuntu install, try the install instructions, and let me know how it goes? I had to take the nfdump out of the install instructions as the version it downloads is said to not work. So before I started anything else, I downloaded the main dependencies, then installed nfdump as said here, and then nfsen as provided. I had to make the following folders: mkdir /var/nfdump mkdir /var/nfdump/profiles-data mkdir /var/nfdump/profiles-data/live mkdir /var/nfdump/profiles-data/live/source, and then allow www:data, and then build this folder as well: /var/www/html/nfsen-ng/backend/datasources Otherwise I get warnings that the following folders do not exist. Sounds like this works fine on Debian, but I have tried over 8 times now on Ubuntu 22.04 and got nothing. If this is to be unsupported by 22.04, i just need to know is all - thank you!

[FontouraAbreu]

@davidshirk80 -

root@un-machine:~/nfdump-1.7.3# nfdump -V nfdump: Version: 1.7.3-release Date: Fri 09 Feb 2024 11:35:59 AM CST

Mine:

root@nfsen-ng:/var/www/html/nfsen-ng/backend# nfdump -V
nfdump: Version: NSEL-NEL1.7.1

---

settings.php has tried both with and without live to same result

'nfdump' => array(
   'binary' => '/usr/bin/nfdump',
   'profiles-data' => '/var/nfdump/profiles-data/live',
   'profile' => 'live',
   'max-processes' => 1, // maximum number of concurrently running nfdump processes

Mine are:

'nfdump' => array(
    'binary' => '/usr/bin/nfdump',
    'profiles-data' => '/var/nfdump/profiles-data/',
    'profile' => 'live',
    'max-processes' => 1, // maximum number of concurrently running nfdump processes

---

root@un-machine:~# ls -la /var/www/html/nfsen-ng/backend/datasources/data/ total 8 drwxrwxrwx 2 root root 4096 Feb 9 11:51 . drwxr-xr-x 3 www-data www-data 4096 Feb 9 11:51 ..

Did you create these manually? I'm didn't need to compile nfdump from source

---

Please send us your /lib/systemd/system/nfdump@.service content.

This is mine:

[Unit]
Description=netflow capture daemon, %I instance
Documentation=man:sfcapd(1)
After=network.target auditd.service
PartOf=nfdump.service

[Service]
Type=forking
EnvironmentFile=/etc/nfdump/%I.conf
ExecStart=/usr/bin/sfcapd -w /var/nfdump/profiles-data/live/source -D -S 1 -B 200000 -P /run/sfcapd.%I.pid $options
PIDFile=/run/sfcapd.%I.pid
KillMode=process
Restart=no

[Install]
WantedBy=multi-user.target

I'm using sfcapd instead of nfcapd because my source sends only sflow instead of netflow.

keep in mind that flags for sfcapd and nfcapd are different

---

Also, send us your /var/www/html/nfsen-ng/backend/settings/settings.php content

---

My /var/www/html/nfsen-ng/backend/datasources/data permissions are:

root@nfsen-ng:/var/www/html/nfsen-ng/backend# ls -la datasources/data
total 111980
drwxr-xr-x 2 www-data www-data    4096 fev 15 10:05 .
drwxr-xr-x 3 www-data www-data    4096 fev  7 13:05 ..
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 123.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 1433.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 1434.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 161.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 162.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 22.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 53.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev  7 13:30 6313.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 13:05 69.rrd
-rw-r--r-- 1 www-data www-data 5457448 fev 15 10:00 80.rrd

---

Maybe another fresh-install can help you?

[davidshirk80]

My NFdump was version 1.6x when downloaded with disto default, and when compiled from phaag was what I posted - will do try it with the version you listed and see how it goes! Will try cli.php without making live folder. I had to create /var/www/html/nfsen-ng/backend/datasources/data/ manually. Will redo box and sent /var/www/html/nfsen-ng/backend/settings/settings.php

Thank you!

[FontouraAbreu]

I had to create /var/www/html/nfsen-ng/backend/datasources/data/ manually.

This path was created automatically for me when I corrected my /etc/nfdump/default.conf file and changed de profiles-path variable in settings.php. Here https://github.com/mbolli/nfsen-ng/issues/17#issuecomment-1932056866

[davidshirk80]

where can I download this version of nfdump? nfdump: Version: NSEL-NEL1.7.1 it it not the distro version or the one phaag has out (which the most recent install is only for ubuntu 18.04. I followed the directions to the letter, and followed the corrections in 17 to the letter. Nothing changed :-/

[davidshirk80]

$nfsen_config = array( 'general' => array( 'ports' => array( 80, 22, 53, ), 'sources' => array( 'router101', ), 'db' => 'RRD', 'processor' => 'NfDump', ), 'frontend' => array( 'reload_interval' => 60, 'defaults' => array( 'view' => 'graphs', // graphs, flows, statistics 'graphs' => array( 'display' => 'sources', // sources, protocols, ports 'datatype' => 'flows', // flows, packets, bytes 'protocols' => array('any'), // any, tcp, udp, icmp, others (multiple possible if display=protocols) ), 'flows' => array( 'limit' => 50, ), 'statistics' => array( 'order_by' => 'bytes', ), ), ), 'nfdump' => array( 'binary' => '/usr/bin/nfdump', 'profiles-data' => '/var/nfdump/profiles-data/live', 'profile' => 'live', 'max-processes' => 1, // maximum number of concurrently running nfdump processes ), 'db' => array( 'Akumuli' => array( //'host' => 'localhost', //'port' => 8282, ), 'RRD' => array() ), 'log' => array( 'priority' => LOG_INFO, // LOG_DEBUG is very talkative! ) );

[FontouraAbreu]

@davidshirk80 - I'll try and install this branch: https://github.com/mbolli/nfsen-ng/tree/prep-php-82 as @mbolli requested earlier and will keep track of any changes, updates or downloads that i've made to make it work. I suggest you try it as well. I'll keep in touch

[FontouraAbreu]

getting this though :-/ Feb 12 16:36:10 machine systemd[1]: Started nfsen-ng. Feb 12 16:36:10 machine su[16390]: (to www-data) root on none Feb 12 16:36:10 machine su[16390]: pam_unix(su-l:session): session opened for user www-data(uid=33) by (uid=0) Feb 12 16:36:11 machine su[16390]: pam_unix(su-l:session): session closed for user www-data

getting the same in this new installation. Not sure why this would happened, but it seems to be the only difference between my last installation. @mbolli any hints?

[FontouraAbreu]

In my /var/log/apache2/error.log I see:

[Wed Feb 21 16:23:50.976407 2024] [php:warn] [pid 14244] [client 172.16.0.205:42574] PHP Warning:  include_once(/var/www/html/nfsen-ng/backend/../vendor/autoload.php): Failed to open stream: No such file or directory in /var/www/html/nfsen-ng/backend/index.php on line 3, referer: http://172.16.0.163/nfsen-ng/frontend/
[Wed Feb 21 16:23:50.976603 2024] [php:warn] [pid 14244] [client 172.16.0.205:42574] PHP Warning:  include_once(): Failed opening '/var/www/html/nfsen-ng/backend/../vendor/autoload.php' for inclusion (include_path='.:/usr/share/php') in /var/www/html/nfsen-ng/backend/index.php on line 3, referer: http://172.16.0.163/nfsen-ng/frontend/
[Wed Feb 21 16:23:50.977001 2024] [php:error] [pid 14244] [client 172.16.0.205:42574] PHP Fatal error:  Uncaught Error: Class "mbolli\\nfsen_ng\\api\\Api" not found in /var/www/html/nfsen-ng/backend/index.php:12\nStack trace:\n#0 {main}\n  thrown in /var/www/html/nfsen-ng/backend/index.php on line 12, referer: http://172.16.0.163/nfsen-ng/frontend/
[Wed Feb 21 17:03:49.485012 2024] [mpm_prefork:notice] [pid 14240] AH00170: caught SIGWINCH, shutting down gracefully

---

In my nfsen-ng dashboard I see:

My source is not being displayed either.

---

When trying to filter something I get:

---

My nfcapd files have content:

root@nfsen-ng:/var/www/html/nfsen-ng/backend# ls -l /var/nfdump/profiles-data/live/source/2024/02/22/ | head
total 28972
-rw-r--r-- 1 root root 260458 Feb 21 21:05 nfcapd.202402220000
-rw-r--r-- 1 root root 244595 Feb 21 21:10 nfcapd.202402220005
-rw-r--r-- 1 root root 237489 Feb 21 21:15 nfcapd.202402220010
-rw-r--r-- 1 root root 239413 Feb 21 21:20 nfcapd.202402220015
-rw-r--r-- 1 root root 231333 Feb 21 21:25 nfcapd.202402220020
-rw-r--r-- 1 root root 239906 Feb 21 21:30 nfcapd.202402220025
-rw-r--r-- 1 root root 235330 Feb 21 21:35 nfcapd.202402220030
-rw-r--r-- 1 root root 231995 Feb 21 21:40 nfcapd.202402220035
-rw-r--r-- 1 root root 246387 Feb 21 21:45 nfcapd.202402220040

---

This is my settings.php:

$nfsen_config = [
    'general' => [
        'ports' => [
            22, 53,
        ],
        'sources' => [
            'source',
        ],
        'db' => 'RRD',
        'processor' => 'NfDump',
    ],
    'frontend' => [
        'reload_interval' => 60,
        'defaults' => [
            'view' => 'graphs', // graphs, flows, statistics
            'graphs' => [
                'display' => 'sources', // sources, protocols, ports
                'datatype' => 'flows', // flows, packets, traffic
                'protocols' => ['any'], // any, tcp, udp, icmp, others (multiple possible if display=protocols)
            ],
            'flows' => [
                'limit' => 50,
            ],
            'statistics' => [
                'order_by' => 'bytes',
            ],
        ],
    ],
    'nfdump' => [
        'binary' => '/usr/bin/nfdump',
        'profiles-data' => '/var/nfdump/profiles-data/',
        'profile' => 'live',
        'max-processes' => 1, // maximum number of concurrently running nfdump processes
    ],

[FontouraAbreu]

After doing systemctl restart nfsen-ng.service I see this in journalctl -ru nfsen-ng.service:

Feb 22 10:23:16 nfsen-ng su[1998]: pam_unix(su-l:session): session closed for user www-data
Feb 22 10:23:16 nfsen-ng su[1998]: pam_unix(su-l:session): session opened for user www-data(uid=33) by (uid=0)
Feb 22 10:23:16 nfsen-ng su[1998]: (to www-data) root on none
Feb 22 10:23:16 nfsen-ng systemd[1]: Started nfsen-ng.service - nfsen-ng.
Feb 22 10:23:16 nfsen-ng systemd[1]: Stopped nfsen-ng.service - nfsen-ng.
Feb 22 10:23:16 nfsen-ng systemd[1]: nfsen-ng.service: Deactivated successfully.
Feb 22 10:23:16 nfsen-ng su[1992]: pam_unix(su-l:session): session closed for user www-data
Feb 22 10:23:15 nfsen-ng su[1992]: pam_unix(su-l:session): session opened for user www-data(uid=33) by (uid=0)
Feb 22 10:23:15 nfsen-ng su[1992]: (to www-data) root on none
Feb 22 10:23:15 nfsen-ng systemd[1]: Stopping nfsen-ng.service - nfsen-ng...

[FontouraAbreu]

Alright, as in this comment something magical happened.

I've been trying to solve the issue with the graphs not showing since last thursday but didn't manage to solve it, then left it to try again today(monday).

Accessing nfsen-ng web-page I noticed the graphs were working and apparently normal.

Looking at journalctl --since "2024-02-22" -g "nfsen-ng" for error logs I didn't find anything that would indicate something wrong

I still believe that the application need some time or data collected to start working and showing graphs, what could be causing this behavior?

Obs.: Nfdump filter wasn't working either

[FontouraAbreu]

@davidshirk80 - Arrived at office today and tried something:

Accessing the nfsen-ng web page using my office browser today , which is were nothing was working last week, still didn't work.

While at home, in another machine and browser, it worked fine. I believe this is related to a browser session error, so I simply cleared cookies for the website and everything started working again.

Maybe try that and see what happens

@mbolli I'll be adding this probable issue in the readme file I'm working on

[davidshirk80]

Tried installing with the fork provided - and keep getting held up here: user@user-test2:~/nfdump-1.7.4$ sudo echo "extension=rrd.so" > /etc/php/8.3/mods-available/rrd.ini -bash: /etc/php/8.3/mods-available/rrd.ini: Permission denied

[mbolli]

@davidshirk80 as is stated in the installation instructions, you need to execute these commands as root. With sudo, try vim /etc/php/8.3/mods-available/rrd.ini (or another editor) and paste extension=rrd.so and save