UnveilTech commented 2 months ago

Hi Toni, sorry for posting here a request...

we would like to propose a new feature with the nDPId, a special argument for a .conf file instead of having all options on the command line. we could have something like that:

command: ./nDPId -x /etc/nDPId/nDPId.conf
(-x or -C or whatever you think it should be fine with you)
nDPId.conf file including options: interface=eth0 logfile=/var/log/nDPId/ndpid.log socket=/tmp/ndpid-collector.sock or 127.0.0.1:7000 background=yes protofile=/etc/nDPId/myprotos.txt etc...

it could also be nice to have a special compilation option (ex: -DCONF_FILE=/etc/nDPId/nDPId.conf)

do you think it makes sense to you ?

have a nice day, Fred

utoni commented 2 months ago

I had this already in my mind. But I thought nobody would be interested in having this feature. Will do ASAP.

UnveilTech commented 2 months ago

Toni, it will be readable and useful :o) then all popular products use that way with settings...

not an emmergency, just a "nice-to-have".

bye Fred

UnveilTech commented 2 weeks ago

Toni,

giving a try, we keep you posted once compiled and started...

bye Fred

utoni commented 2 weeks ago

Thanks! Hoping to get some feedback from you guys! (positive and negative) :smile_cat:

UnveilTech commented 2 weeks ago

compiled, launched, it seems all is ok........................... for the moment. -R and -f work fine.

we keep it running in our lab for a while, we keep you posted. anyway, thanks a lot for your work !!!

bye Fred

UnveilTech commented 2 weeks ago

Hi Toni,

we have detected a strange behaviour with the conf file, let me explain...

we have used the 'ndpid.conf.example' (renamed ndpid.conf) you provide with the original source but those 3 options seems to act not correctly, we did not use them with previous builds: _#internal = true

external = true

analysis = true_

we kept 'internal' and 'external' commented so it means not enabled in the settings. but the option 'analysis' seems not working properly, actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent, indicating many MB up to many GB with protos (ex: ICMP, DNS, ...). we have not those strange values with the command line, surely because those options are 'false' by default.

by setting those 3 options to false we now get normal values.

so our advise is to modify the file 'ndpid.conf.example' and set 'false' by default with those 3 options or with a warning that 'people must take blablabla...' especialy with the 'internal'.

except that point, latest build works fine. :o)

bye Fred

UnveilTech commented 2 weeks ago

by "actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent", we mean values are always the same and sizes are increasing with incoherent value...

UnveilTech commented 2 weeks ago

Toni, here is the settings working fine with us:

[general] ... internal=false external=false compression=true analysis=false

[tuning] max-flows-per-thread=2048 max-idle-flows-per-thread=64 max-reader-threads=10 daemon-status-interval=600000000

memory-profiling-log-interval=5

compression-scan-interval=20000000 compression-flow-inactivity=30000000 flow-scan-interval=10000000 generic-max-idle-time=600000000 icmp-max-idle-time=120000000 tcp-max-idle-time=180000000 udp-max-idle-time=7440000000 tcp-max-post-end-flow-time=120000000 max-packets-per-flow-to-send=15 max-packets-per-flow-to-process=32 max-packets-per-flow-to-analyse=32 error-event-threshold-n=16 error-event-threshold-time=10000000

[ndpi] packets_limit_per_flow=32 flow.direction_detection=enable flow.track_payload=disable tcp_ack_payload_heuristic=disable fully_encrypted_heuristic=enable libgcrypt.init=1 dpi.compute_entropy=1 fpc=enable # you set it 'disable' dpi.guess_on_giveup=0x03 flow_risk_lists.load=1 flow_risk.crawler_bot.list.load=1 log.level=0

[protos] tls.certificate_expiration_threshold=30 # we set it back to default tls.application_blocks_tracking=disable # we set it back to default stun.max_packets_extra_dissection=25 # we set it back to default

bye Fred

utoni commented 2 weeks ago

Hi Fred

we have used the 'ndpid.conf.example' (renamed ndpid.conf) you provide with the original source but those 3 options seems to act not correctly, we did not use them with previous builds: #internal = true #external = true analysis = true

we kept 'internal' and 'external' commented so it means not enabled in the settings. but the option 'analysis' seems not working properly, actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent, indicating many MB up to many GB with protos (ex: ICMP, DNS, ...). we have not those strange values with the command line, surely because those options are 'false' by default.

If the analysis mode is enabled, the daemon will consume much more heap memory. The reason behind that is that it needs additional memory necessary to store some statistical values until max-packets-per-flow-to-analyse reached before firing an analyse event which contains all those statistical values. The heap memory used for that is free'd afterwards.

by "actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent", we mean values are always the same and sizes are increasing with incoherent value...

That's something I do not understand. Could you please elaborate this more (or provide an example)?

utoni commented 2 weeks ago

But I'll comment out analysis = true, which will disable the analysis completely and add some comments about memory consumption.

UnveilTech commented 2 weeks ago

by "actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent", we mean values are always the same and sizes are increasing with incoherent value...

That's something I do not understand. Could you please elaborate this more (or provide an example)?

Ya I know, not easy to understand, we spent hours to see the issue and tried to identify where the problem could be. let me summarize what we saw:

using your .conf with default values we got results with many strange values
we first have commented the part [protos], same (wrong) results
then we have commented [ndpi], then [tuning] ([tuning] seem to be default values), same result
finally, set to false 'internal', 'external' and 'analysis', here results were correct

what I mean by 'corect' is that the key 'flow_id' to follow detected/update/end-idle is correct to get the hostname and the final size (case: https://github.com/utoni/nDPId/issues/39). but with the default values of the ndpid.conf, the same key 'flow_id' was (many) often the same giving incoherent results as indicated. we suspect the cause is the key 'analysis' that is, if I'm right, a recent option with the libnDPI

right now, we only use that: [general] ... internal=false external=false compression=true analysis=false

other sections are commented from our side.

Fred

UnveilTech commented 2 weeks ago

more infos:

we have not found all options in the nDPId.c like 'fpc' in the section [ndpi] regarding 'fpc', you use 'disable' but 'libnDPI/doc/configuration_parameters.md' use 'enable'

utoni commented 2 weeks ago

I've just tested the analyse event within nDPId-test.c. It works on my side.

You may test this as well on your side by changing:

set_cmdarg_boolean(&nDPId_options.enable_data_analysis, 1);

to

set_cmdarg_boolean(&nDPId_options.enable_data_analysis, 0);

in nDPId-test.c:1671.

You may then run ./test/run_tests.sh ./libnDPI ./path/to/nDPId-test and use git diff afterwards to take a closer look what test results have been changed. Only analyse events should disappear.

we have not found all options in the nDPId.c like 'fpc' in the section [ndpi] regarding 'fpc', you use 'disable' but 'libnDPI/doc/configuration_parameters.md' use 'enable'

That's something I'll need to take a closer look. But from my understanding, fpc provides a faster packet classification, but may result in false-positives in some corner cases.

UnveilTech commented 2 weeks ago

Toni, we have set back your original config, have a look: {"flow_event_id":4,"flow_event_name":"update","thread_id":1,"packet_id":59620,"source":"enp4s0","alias":"ATiBox","flow_id":30,"flow_state":"finished","flow_src_packets_processed":12,"flow_dst_packets_processed":0,"flow_first_seen":1729254260350023,"flow_src_last_pkt_time":1729254319506703,"flow_dst_last_pkt_time":1729254260350023,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":36,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":36,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":432,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254330000712,"l3_proto":"ip4","src_ip":"80.12.193.50","dst_ip":"192.168.4.6","l4_proto":"icmp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}

update: 1729254260350023, 30, ICMP, 432 <- our special line in the our log file, we write the flow_event_name, date, flowid, proto and size

{"flow_event_id":7,"flow_event_name":"detected","thread_id":1,"packet_id":59676,"source":"enp4s0","alias":"ATiBox","flow_id":2096,"flow_state":"info","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1729254330592998,"flow_src_last_pkt_time":1729254330592998,"flow_dst_last_pkt_time":1729254330592998,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":8,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":8,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":8,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254330592998,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"144.76.59.84","l4_proto":"icmp","ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}

detected: 1729254330592998, 2096, ICMP, 8

{"flow_event_id":7,"flow_event_name":"detected","thread_id":1,"packet_id":59916,"source":"enp4s0","alias":"ATiBox","flow_id":2150,"flow_state":"info","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1729254332697872,"flow_src_last_pkt_time":1729254332697872,"flow_dst_last_pkt_time":1729254332697872,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":36,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":36,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":36,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254332697872,"l3_proto":"ip4","src_ip":"193.252.159.45","dst_ip":"192.168.4.6","l4_proto":"icmp","ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network","entropy":4.450212}} detected: 1729254332697872, 2150, ICMP, 36

{"flow_event_id":4,"flow_event_name":"update","thread_id":0,"packet_id":60742,"source":"enp4s0","alias":"ATiBox","flow_id":236,"flow_state":"finished","flow_src_packets_processed":2,"flow_dst_packets_processed":0,"flow_first_seen":1729254266339156,"flow_src_last_pkt_time":1729254284862241,"flow_dst_last_pkt_time":1729254266339156,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":69,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":69,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":138,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254339725963,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"1.1.1.1","l4_proto":"icmp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}

update: 1729254266339156, 236, ICMP, 138

{"flow_event_id":4,"flow_event_name":"update","thread_id":1,"packet_id":60755,"source":"enp4s0","alias":"ATiBox","flow_id":1401,"flow_state":"finished","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1729254306498409,"flow_src_last_pkt_time":1729254306498409,"flow_dst_last_pkt_time":1729254306498409,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":148,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":148,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":148,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254339822064,"l3_proto":"ip4","src_ip":"81.253.129.137","dst_ip":"192.168.4.6","l4_proto":"icmp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}

update: 1729254306498409, 1401, ICMP, 148

it seems the 'flow_id' becomes crazy using very old values when it should and id 2,000+ so after a while, a proto is with a strange (too big) size value.

'fpc' could generate some false-positives, ok but it seems not used by the nDPId.c.

we'll comment option by option in order to find which one is causin the issue, keep you posted...

bye Fred

UnveilTech commented 2 weeks ago

Toni, have a look:

ICMP: 1.3MB with 2 hits in the last 7 seconds. something is wrong here :o)

Fred

utoni commented 2 weeks ago

it seems the 'flow_id' becomes crazy using very old values when it should and id 2,000+ so after a while, a proto is with a strange (too big) size value.

I guess that I understood your problem now. The behavior you're mentioned regarding the flow_id is totally legit. Let's take a close look. The event name that contains that low/old flow_id is called update. According to the documentation:

update: inform nDPIsrvd or other apps about a long-lasting flow, whose detection was finished a long time ago but is still active

'fpc' could generate some false-positives, ok but it seems not used by the nDPId.c.

That's correct. I am currently testing the fpc behavior.

utoni commented 2 weeks ago

Toni, have a look:

ICMP: 1.3MB with 2 hits in the last 7 seconds. something is wrong here :o)

Regarding the ICMP flow (also from the README.md)

Other protocols e.g. ICMP/ICMPv6 lack relevance for DPI, thus nDPId does not distinguish between different ICMP/ICMPv6 flows coming from the same host. This saves memory and performance, but might change in the future.

UnveilTech commented 2 weeks ago

I do not agree, we get ICMP and ICMPv6 from the nDPId, have a look:

update: inform nDPIsrvd or other apps about a long-lasting flow, whose detection was finished a long time ago but is still active

long time ago could mean 2+ hours ago ? hmmm, I'm not sure about that ;o)

Fred

UnveilTech commented 2 weeks ago

Toni, additional tests done, we suspect the issue is in that part: [protos] tls.certificate_expiration_threshold = 7 tls.application_blocks_tracking = enable stun.max_packets_extra_dissection = 8

original values from libnDPI (libnDPI/doc/configuration_parameters.md) are like that: [protos] tls.certificate_expiration_threshold = 30 tls.application_blocks_tracking = disable stun.max_packets_extra_dissection = 25

using libnDPI values ([protos]), results look correct but we keep a look on data...

Fred

utoni commented 2 weeks ago

long time ago could mean 2+ hours ago ? hmmm, I'm not sure about that ;o)

An ICMP flow is considered active as long as there is still data on the wire which does belong to that flow. It will time out after icmp-max-idle-time if there was no data seen on the wire during that time. On timeout an idle event will be generated.

original values from libnDPI (libnDPI/doc/configuration_parameters.md) are like that: [protos] tls.certificate_expiration_threshold = 30 tls.application_blocks_tracking = disable stun.max_packets_extra_dissection = 25

I think I'll restore the config file to the default nDPI values.

utoni commented 2 weeks ago

I'll reopen the issue although it might be better to split the issue into two.

utoni commented 2 weeks ago

You may now use branch https://github.com/utoni/nDPId/tree/tmp and run ./test/run_config_tests.sh to test certain configurations placed in ./test/configs.

UnveilTech commented 1 week ago

hello Toni,

here is the test:

does it look ok ?

bye Fred

utoni commented 1 week ago

It does. Closing the issue. Please open a new one if your problem persists.

utoni / nDPId

Conf file #41

external = true

memory-profiling-log-interval=5