Do you have any plan to support FreeBSD ?

[fateme81]

I'm trying to build this on FreeBSD and I get whole lot of errors. Can you help me on this ?

[utoni]

Yes. BSD and Mac support will be part of the next release. This is the last milestone before 1.6.

[fateme81]

Can you estimate when it's gonna be finished ? I would like to help you on this if it's required.

[utoni]

Two things that decreases development/testing time: Could you paste the build errors you're getting on BSD here? Another thing is the testing of the reworked event I/O (nio.h and nio.c). It supports poll() which is vital for BSD and Mac. I am currently working on the integration into nDPId, nDPIsrvd and the (C) dependencies. After that is finished, I'll need someone who can test this on both platforms.

[fateme81]

The output of gmake :

Consolidate compiler generated dependencies of target nDPIsrvd-analysed
[ 10%] Built target nDPIsrvd-analysed
Consolidate compiler generated dependencies of target nDPIsrvd-json-dump
[ 17%] Built target nDPIsrvd-json-dump
[ 46%] Built target libnDPI
Consolidate compiler generated dependencies of target nDPIsrvd-captured
[ 57%] Built target nDPIsrvd-captured
[ 60%] Building C object CMakeFiles/nDPIsrvd-collectd.dir/examples/c-collectd/c-collectd.c.o
/home/fatemeh/git/ndpid/examples/c-collectd/c-collectd.c:9:10: fatal error: 'sys/epoll.h' file not found
#include <sys/epoll.h>
         ^~~~~~~~~~~~~
1 error generated.
gmake[2]: *** [CMakeFiles/nDPIsrvd-collectd.dir/build.make:76: CMakeFiles/nDPIsrvd-collectd.dir/examples/c-collectd/c-collectd.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:232: CMakeFiles/nDPIsrvd-collectd.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

[fateme81]

And I would like to help you on testing this on FreeBSD.

[utoni]

This is the expected result and exactly what I am working on at the moment. When I started with nDPId, I initially thought that this toolkit would be only relevant for Linux users. So I decided to use epoll() for event I/O, which is only available on Linux. No Win, no Mac, no BSD. But things changed and therefor I need to adapt and support portable event I/O like poll().

[utoni]

You may now try the add/event-io-abstraction branch. The newly added event I/O abstraction layer should make it possible to use nDPId and nDPIsrvd within BSD and Mac. Please report any errors, configure/build/runtime failures included.

Cheers!

[fateme81]

First off all I want to thank you for time you spent on replacing Epoll. I tried to build nDPId on FreeBSD and this is the output :

In file included from /home/fatemeh/git/nDPId/nDPId.c:5:
/usr/include/net/ethernet.h:61:2: error: unknown type name 'u_char'; did you mean 'char'?
        u_char  ether_dhost[ETHER_ADDR_LEN];
        ^
/usr/include/net/ethernet.h:62:2: error: unknown type name 'u_char'; did you mean 'char'?
        u_char  ether_shost[ETHER_ADDR_LEN];
        ^
/usr/include/net/ethernet.h:63:2: error: unknown type name 'u_short'; did you mean 'short'?
        u_short ether_type;
        ^
/usr/include/net/ethernet.h:70:2: error: unknown type name 'u_char'; did you mean 'char'?
        u_char octet[ETHER_ADDR_LEN];
        ^
/usr/include/net/ethernet.h:82:2: error: unknown type name 'uint8_t'
        uint8_t evl_dhost[ETHER_ADDR_LEN];
        ^
/usr/include/net/ethernet.h:83:2: error: unknown type name 'uint8_t'
        uint8_t evl_shost[ETHER_ADDR_LEN];
        ^
/home/fatemeh/git/nDPId/nDPId.c:20:10: fatal error: 'sys/signalfd.h' file not found
#include <sys/signalfd.h>
         ^~~~~~~~~~~~~~~~
7 errors generated.
gmake[2]: *** [CMakeFiles/nDPId.dir/build.make:76: CMakeFiles/nDPId.dir/nDPId.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:102: CMakeFiles/nDPId.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

[utoni]

Dang. I totally forgot that signalfd() is Linux specific. I'll find a way around (probably falling back to Posix signal() handler).

The other errors do not seem to be any obstacle to fix them. Just #include.

[fateme81]

Just let me know if any help is needed. Good luck:)

[fateme81]

Just my daily checking for new updates. so excited to finally can work with nDPId on Freebsd.

[utoni]

Thanks for your interest!

You can watch the progress in https://github.com/utoni/nDPId/tree/tmp I am currently fixing some other issues related to non-POSIX features.

[utoni]

Please test PR #30 and give me some feedback if possible. :)

[fateme81]

Hey, Thanks for notifying me :) I tried to build and here is the output :

config.status: creating Makefile
config.status: creating example/Makefile
config.status: creating example/Makefile.dpdk
config.status: creating tests/Makefile
config.status: creating tests/unit/Makefile
config.status: creating tests/performance/Makefile
config.status: creating tests/dga/Makefile
config.status: creating rrdtool/Makefile
config.status: creating influxdb/Makefile
config.status: creating libndpi.pc
config.status: creating src/include/ndpi_define.h
config.status: creating src/lib/Makefile
config.status: creating fuzz/Makefile
config.status: creating doc/Doxyfile.cfg
config.status: creating tests/do.sh
config.status: creating src/include/ndpi_config.h
config.status: executing depfiles commands
config.status: executing libtool commands
+ /usr/local/bin/gmake s install DESTDIR=/home/fatemeh/git/ndpi2/build/libnDPI
gmake[3]: *** No rule to make target 's'.  Stop.
gmake[2]: *** [CMakeFiles/libnDPI.dir/build.make:92: libnDPI-prefix/src/libnDPI-stamp/libnDPI-configure] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:223: CMakeFiles/libnDPI.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

[utoni]

-DBUILD_NDPI=OFF is required for the moment, because the libnDPI build script needs get ported separately.

[fateme81]

ndpi version : 4.8.0 The output with -DBUILD_NDPI=OFF is :

/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:188:28: error: field has incomplete type 'struct sockaddr_in'
        struct sockaddr_in in;
                           ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:188:16: note: forward declaration of 'struct sockaddr_in'
        struct sockaddr_in in;
               ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:189:29: error: field has incomplete type 'struct sockaddr_in6'
        struct sockaddr_in6 in6;
                            ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:189:16: note: forward declaration of 'struct sockaddr_in6'
        struct sockaddr_in6 in6;
               ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:449:23: error: no member named 'raw' in 'struct nDPIsrvd_address'
        sock->address.raw.sa_family = -1;
        ~~~~~~~~~~~~~ ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:636:18: error: no member named 'raw' in 'struct nDPIsrvd_address'
        address->raw.sa_family = AF_UNIX;
        ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:637:41: error: no member named 'un' in 'struct nDPIsrvd_address'
        address->size = sizeof(address->un);
                               ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:638:31: error: no member named 'un' in 'struct nDPIsrvd_address'
        if (snprintf(address->un.sun_path, sizeof(address->un.sun_path), "%s", destination) <= 0)
                     ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:638:60: error: no member named 'un' in 'struct nDPIsrvd_address'
        if (snprintf(address->un.sun_path, sizeof(address->un.sun_path), "%s", destination) <= 0)
                                                  ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:652:22: error: no member named 'raw' in 'struct nDPIsrvd_address'
            address->raw.sa_family = AF_INET;
            ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:653:45: error: no member named 'in' in 'struct nDPIsrvd_address'
            address->size = sizeof(address->in);
                                   ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:654:22: error: no member named 'in' in 'struct nDPIsrvd_address'
            address->in.sin_port = htons(atoi(last_colon + 1));
            ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:655:35: error: no member named 'in' in 'struct nDPIsrvd_address'
            sock_addr = &address->in.sin_addr;
                         ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:664:22: error: no member named 'raw' in 'struct nDPIsrvd_address'
            address->raw.sa_family = AF_INET6;
            ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:665:45: error: no member named 'in6' in 'struct nDPIsrvd_address'
            address->size = sizeof(address->in6);
                                   ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:666:22: error: no member named 'in6' in 'struct nDPIsrvd_address'
            address->in6.sin6_port = htons(atoi(last_colon + 1));
            ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:667:35: error: no member named 'in6' in 'struct nDPIsrvd_address'
            sock_addr = &address->in6.sin6_addr;
                         ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:688:32: error: no member named 'raw' in 'struct nDPIsrvd_address'
        if (inet_pton(address->raw.sa_family, addr_buf, sock_addr) != 1)
                      ~~~~~~~  ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:699:37: error: no member named 'raw' in 'struct nDPIsrvd_address'
    sock->fd = socket(sock->address.raw.sa_family, SOCK_STREAM, 0);
                      ~~~~~~~~~~~~~ ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:706:42: error: no member named 'raw' in 'struct nDPIsrvd_address'
    if (connect(sock->fd, &sock->address.raw, sock->address.size) != 0)
                           ~~~~~~~~~~~~~ ^
/home/fatemeh/git/ndpid/nDPIsrvd.c:67:32: error: field has incomplete type 'struct sockaddr_in'
            struct sockaddr_in peer;
                               ^
/home/fatemeh/git/ndpid/dependencies/nDPIsrvd.h:188:16: note: forward declaration of 'struct sockaddr_in'
        struct sockaddr_in in;
               ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
gmake[2]: *** [CMakeFiles/nDPIsrvd.dir/build.make:76: CMakeFiles/nDPIsrvd.dir/nDPIsrvd.c.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:115: CMakeFiles/nDPIsrvd.dir/all] Error 2
gmake: *** [Makefile:156: all] Error 2

[utoni]

Commit 577cf6992cbdecea1f7837e449bbfe6c9b91a402 should address this issue.

[fateme81]

ANDDDDDDDDDDDDDDDDDDDDD FINALYYYYY:) I just tried again and building nDPId on FreeBSD was completely successful. So excited to run and use it. Thank u so much:))))))))))))))))

[fateme81]

Well I just tried to run it using following command : ./nDPId -u root -g root -l -i em0 -c /tmp/listen.socket

and the output : nDPId [error]: Could not get netmask for pcap device em0: Invalid argument Do you know why this happens?

[utoni]

This issue is most likely caused by ioctl(sock, SIOCAIFADDR, &ifr), but according to BSD manpages this should technically work. Need to investigate.

[utoni]

I've used a wrong ioctl() request id to retrieve address + netmask. Completly reworked w/o using ioctl()s. See commit 6df7247ceca7c30169c706cd8a8eb3ec522e8d0c.

[utoni]

Any updates @fateme81 ?

// You may now use the main branch, which should work on *BSD and will only receive bugfixes till the 1.6 release is done.

[fateme81]

I'm sorry for the delay i did not have access to my system for a while. I ran nDPId again and i did not catch any errors about interface. It's started to work without problem however i did not try making a listening socket and send logs to that socket. I hope that work without problem too. I will notify you in case of any new errors. Thanks.

[utoni]

I am closing this issue now. If there are any issues, please open a new issue. :)

[fateme81]

After working with ndpid i discovered that some logs are missing and the reason is some threads can not connect to nDPIsrvd Collector at my unix socket. The error is as following :

nDPId [error]: Thread 10: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused
nDPId [error]: Thread 11: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused
nDPId [error]: Thread 15: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused
nDPId [error]: Thread 14: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused
nDPId [error]: Thread 13: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused
nDPId [error]: Thread 12: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused
nDPId [error]: Thread 9: Could not connect to nDPIsrvd Collector at /tmp/listen.socket, will try again later. Error: Connection refused

The command i use to make a socket : nc -lkU /tmp/listen.socket

can you help me fix this

[fateme81]

I have some logs randomly but not all of them!

[utoni]

The issue is netcat which may only handle one connection at a time. To make nDPId work together with netcat you need to force nDPId to use only one thread e.g.: sudo nDPId -o max-reader-threads=1