*_tot_l4_payload_len seem not correct

[UnveilTech]

hello,

we're testing a 1Mb file and we don't find the correct downloaded size in the log. are we wrong or is that normal ?

test file: https://proof.ovh.net/files/1Mb.dat

results in the nDPId log file: 01186{"flow_event_id":7,"flow_event_name":"detected","thread_id":3,"packet_id":26512916,"source":"enp1s0","alias":"firewall","flow_id":379423,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":2,"flow_first_seen":1723646877940304,"flow_src_last_pkt_time":1723646877973113,"flow_dst_last_pkt_time":1723646877965159,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723646877973113,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":25067,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"851b45b6dd64c8bf7eef883a3bfb129c","ja3s":"","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01276{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":26512934,"source":"enp1s0","alias":"firewall","flow_id":379423,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":4,"flow_first_seen":1723646877940304,"flow_src_last_pkt_time":1723646877973113,"flow_dst_last_pkt_time":1723646877979420,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723646877979420,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":25067,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"851b45b6dd64c8bf7eef883a3bfb129c","ja3s":"d154fcfa5bb4f0748e1dd1992c681104","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01503{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":26512940,"source":"enp1s0","alias":"firewall","flow_id":379423,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":7,"flow_first_seen":1723646877940304,"flow_src_last_pkt_time":1723646877973113,"flow_dst_last_pkt_time":1723646877981060,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":5257,"midstream":0,"thread_ts_usec":1723646877981060,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":25067,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"851b45b6dd64c8bf7eef883a3bfb129c","ja3s":"d154fcfa5bb4f0748e1dd1992c681104","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:D0:03:95:D1:8B:D5:E6:AC:27:77:36:4D:E5:CA:BB:BF"}}} 01186{"flow_event_id":7,"flow_event_name":"detected","thread_id":9,"packet_id":26512872,"source":"enp1s0","alias":"firewall","flow_id":379424,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":2,"flow_first_seen":1723646877911757,"flow_src_last_pkt_time":1723646877945682,"flow_dst_last_pkt_time":1723646877944924,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723646877945682,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":39073,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"fae04df7936e31c11b49485d6336ba18","ja3s":"","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01276{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":9,"packet_id":26512888,"source":"enp1s0","alias":"firewall","flow_id":379424,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":4,"flow_first_seen":1723646877911757,"flow_src_last_pkt_time":1723646877945682,"flow_dst_last_pkt_time":1723646877952107,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723646877952107,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":39073,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"fae04df7936e31c11b49485d6336ba18","ja3s":"d154fcfa5bb4f0748e1dd1992c681104","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01503{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":9,"packet_id":26512896,"source":"enp1s0","alias":"firewall","flow_id":379424,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":7,"flow_first_seen":1723646877911757,"flow_src_last_pkt_time":1723646877945682,"flow_dst_last_pkt_time":1723646877953882,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":5257,"midstream":0,"thread_ts_usec":1723646877953882,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":39073,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"fae04df7936e31c11b49485d6336ba18","ja3s":"d154fcfa5bb4f0748e1dd1992c681104","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:D0:03:95:D1:8B:D5:E6:AC:27:77:36:4D:E5:CA:BB:BF"}}} 01177{"flow_event_id":7,"flow_event_name":"detected","thread_id":3,"packet_id":26544654,"source":"enp1s0","alias":"firewall","flow_id":381389,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":1,"flow_first_seen":1723647001510009,"flow_src_last_pkt_time":1723647001518401,"flow_dst_last_pkt_time":1723647001516546,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723647001518401,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":14057,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01267{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":26544657,"source":"enp1s0","alias":"firewall","flow_id":381389,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":3,"flow_first_seen":1723647001510009,"flow_src_last_pkt_time":1723647001518401,"flow_dst_last_pkt_time":1723647001524625,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723647001524625,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":14057,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01494{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":26544660,"source":"enp1s0","alias":"firewall","flow_id":381389,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":6,"flow_first_seen":1723647001510009,"flow_src_last_pkt_time":1723647001518401,"flow_dst_last_pkt_time":1723647001526224,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":5285,"midstream":0,"thread_ts_usec":1723647001526224,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":14057,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:D0:03:95:D1:8B:D5:E6:AC:27:77:36:4D:E5:CA:BB:BF"}}}

take care...

bye Fred

[utoni]

Did you get any flow_event_name with value end or idle ? And what states flow_dst_tot_l4_payload_len at that point?

[UnveilTech]

Hi Toni,

we make some tests again and no idle/end in the flow_event_name. only detected/detection-update...

log: 01179{"flow_event_id":7,"flow_event_name":"detected","thread_id":3,"packet_id":5531,"source":"enp1s0","alias":"firewall","flow_id":239,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":2,"flow_first_seen":1723651561250318,"flow_src_last_pkt_time":1723651561268758,"flow_dst_last_pkt_time":1723651561267909,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":2027,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651561268758,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":46757,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"5d914463d2d5fa9a92fc65b837d6a90d","ja3s":"","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01267{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":5537,"source":"enp1s0","alias":"firewall","flow_id":239,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":4,"flow_first_seen":1723651561250318,"flow_src_last_pkt_time":1723651561268758,"flow_dst_last_pkt_time":1723651561274809,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":156,"flow_src_tot_l4_payload_len":2027,"flow_dst_tot_l4_payload_len":156,"midstream":0,"thread_ts_usec":1723651561274809,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":46757,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"5d914463d2d5fa9a92fc65b837d6a90d","ja3s":"d7e12962b60127bdbe4f65f39221f9e8","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01179{"flow_event_id":7,"flow_event_name":"detected","thread_id":9,"packet_id":5520,"source":"enp1s0","alias":"firewall","flow_id":240,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":2,"flow_first_seen":1723651561251502,"flow_src_last_pkt_time":1723651561272623,"flow_dst_last_pkt_time":1723651561270405,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651561272623,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":52533,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"b588034c0b7ad09be625f6ab9390cbd8","ja3s":"","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01267{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":9,"packet_id":5526,"source":"enp1s0","alias":"firewall","flow_id":240,"flow_state":"info","flow_src_packets_processed":4,"flow_dst_packets_processed":4,"flow_first_seen":1723651561251502,"flow_src_last_pkt_time":1723651561272623,"flow_dst_last_pkt_time":1723651561278432,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1448,"flow_dst_max_l4_payload_len":156,"flow_src_tot_l4_payload_len":1931,"flow_dst_tot_l4_payload_len":156,"midstream":0,"thread_ts_usec":1723651561278432,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":52533,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"b588034c0b7ad09be625f6ab9390cbd8","ja3s":"d7e12962b60127bdbe4f65f39221f9e8","ja4":"t00d1516h2_8daaf6152771_02713d6af862","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"GREASE,TLSv1.3,TLSv1.2"}}} 01170{"flow_event_id":7,"flow_event_name":"detected","thread_id":7,"packet_id":5681,"source":"enp1s0","alias":"firewall","flow_id":250,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":1,"flow_first_seen":1723651561726921,"flow_src_last_pkt_time":1723651561735148,"flow_dst_last_pkt_time":1723651561732763,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651561735148,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":39471,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01172{"flow_event_id":7,"flow_event_name":"detected","thread_id":1,"packet_id":32694,"source":"enp1s0","alias":"firewall","flow_id":1346,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":1,"flow_first_seen":1723651622036249,"flow_src_last_pkt_time":1723651622045083,"flow_dst_last_pkt_time":1723651622042692,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651622045083,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":43725,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01262{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":1,"packet_id":32695,"source":"enp1s0","alias":"firewall","flow_id":1346,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":2,"flow_first_seen":1723651622036249,"flow_src_last_pkt_time":1723651622045083,"flow_dst_last_pkt_time":1723651622051396,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723651622051396,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":43725,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01489{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":1,"packet_id":32699,"source":"enp1s0","alias":"firewall","flow_id":1346,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":6,"flow_first_seen":1723651622036249,"flow_src_last_pkt_time":1723651622045083,"flow_dst_last_pkt_time":1723651622053187,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":5285,"midstream":0,"thread_ts_usec":1723651622053187,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":43725,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:00744{"flow_event_id":1,"flow_event_name":"new","thread_id":4,"packet_id":32110,"source":"enp1s0","alias":"firewall","flow_id":1350,"flow_state":"info","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1723651621971010,"flow_src_last_pkt_time":1723651621971010,"flow_dst_last_pkt_time":1723651621971010,"flow_idle_time":200000000,"flow_src_min_l4_payload_len":31,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":31,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":31,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651621971010,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"1.1.1.1","src_port":56138,"dst_port":53,"l4_proto":"udp","flow_datalink":1,"flow_max_packets":15} 01172{"flow_event_id":7,"flow_event_name":"detected","thread_id":5,"packet_id":70739,"source":"enp1s0","alias":"firewall","flow_id":2322,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":1,"flow_first_seen":1723651681348706,"flow_src_last_pkt_time":1723651681357196,"flow_dst_last_pkt_time":1723651681355016,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651681357196,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":36689,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01262{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":5,"packet_id":70741,"source":"enp1s0","alias":"firewall","flow_id":2322,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":3,"flow_first_seen":1723651681348706,"flow_src_last_pkt_time":1723651681357196,"flow_dst_last_pkt_time":1723651681363819,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723651681363819,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":36689,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01489{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":5,"packet_id":70744,"source":"enp1s0","alias":"firewall","flow_id":2322,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":6,"flow_first_seen":1723651681348706,"flow_src_last_pkt_time":1723651681357196,"flow_dst_last_pkt_time":1723651681365106,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":5285,"midstream":0,"thread_ts_usec":1723651681365106,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":36689,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:D0:03:95:D1:8B:D5:E6:AC:27:77:36:4D:E5:CA:BB:BF"}}} 01263{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":106762,"source":"enp1s0","alias":"firewall","flow_id":3249,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":3,"flow_first_seen":1723651741739439,"flow_src_last_pkt_time":1723651741747896,"flow_dst_last_pkt_time":1723651741754612,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723651741754612,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":60797,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01490{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":3,"packet_id":106765,"source":"enp1s0","alias":"firewall","flow_id":3249,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":6,"flow_first_seen":1723651741739439,"flow_src_last_pkt_time":1723651741747896,"flow_dst_last_pkt_time":1723651741756017,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":5285,"midstream":0,"thread_ts_usec":1723651741756017,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":60797,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:D0:03:95:D1:8B:D5:E6:AC:27:77:36:4D:E5:CA:BB:BF"}}} 01173{"flow_event_id":7,"flow_event_name":"detected","thread_id":1,"packet_id":136963,"source":"enp1s0","alias":"firewall","flow_id":4240,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":1,"flow_first_seen":1723651802038143,"flow_src_last_pkt_time":1723651802048016,"flow_dst_last_pkt_time":1723651802046417,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1723651802048016,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":12655,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_NULL_WITH_NULL_NULL","advertised_alpns":"h2,http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01263{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":1,"packet_id":137134,"source":"enp1s0","alias":"firewall","flow_id":4240,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":3,"flow_first_seen":1723651802038143,"flow_src_last_pkt_time":1723651802048016,"flow_dst_last_pkt_time":1723651802055387,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":1448,"midstream":0,"thread_ts_usec":1723651802055387,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":12655,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2"}}} 01490{"flow_event_id":8,"flow_event_name":"detection-update","thread_id":1,"packet_id":137141,"source":"enp1s0","alias":"firewall","flow_id":4240,"flow_state":"info","flow_src_packets_processed":3,"flow_dst_packets_processed":6,"flow_first_seen":1723651802038143,"flow_src_last_pkt_time":1723651802048016,"flow_dst_last_pkt_time":1723651802056994,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":517,"flow_dst_max_l4_payload_len":1448,"flow_src_tot_l4_payload_len":517,"flow_dst_tot_l4_payload_len":5285,"midstream":0,"thread_ts_usec":1723651802056994,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"141.95.207.211","src_port":12655,"dst_port":443,"l4_proto":"tcp","ndpi": {"confidence": {"6":"DPI"},"proto":"TLS","proto_id":"91","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":1,"breed":"Safe","category_id":5,"category":"Web","hostname":"proof.ovh.net","tls": {"version":"TLSv1.2","server_names":"rbx.proof.ovh.net,proof.ovh.net","ja3":"f436b9416f37d134cadd04886327d3e8","ja3s":"263c859c5391203d774bc0599793d915","ja4":"t13d3113h2_e8f1e7e78f70_10734c531abe","unsafe_cipher":0,"cipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","issuerDN":"C=AT, O=ZeroSSL, CN=ZeroSSL RSA Domain Secure Site CA","subjectDN":"CN=rbx.proof.ovh.net","advertised_alpns":"h2,http\/1.1","negotiated_alpn":"http\/1.1","tls_supported_versions":"TLSv1.3,TLSv1.2","fingerprint":"2A:85:64:9B:D0:03:95:D1:8B:D5:E6:AC:27:77:36:4D:E5:CA:BB:BF"}}}

Fred

[utoni]

Seems like a bug. Need to investigate.

[UnveilTech]

Toni,

we did more tests by filtering on "end": 00970{"flow_event_id":2,"flow_event_name":"end","thread_id":1,"packet_id":1048519,"source":"enp1s0","alias":"firewall","flow_id":16624,"flow_state":"finished","flow_src_packets_processed":39577,"flow_dst_packets_processed":88188,"flow_first_seen":1723652592447462,"flow_src_last_pkt_time":1723653691403990,"flow_dst_last_pkt_time":1723653691406740,"flow_idle_time":7580000000,"flow_src_min_l4_payload_len":0,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":1396,"flow_dst_max_l4_payload_len":1396,"flow_src_tot_l4_payload_len":1345955,"flow_dst_tot_l4_payload_len":121587758,"midstream":0,"thread_ts_usec":1723653820454099,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"173.194.190.73","src_port":45915,"dst_port":443,"l4_proto":"tcp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"TLS.YouTube","proto_id":"91.124","proto_by_ip":"Google","proto_by_ip_id":126,"encrypted":1,"breed":"Fun","category_id":1,"category":"Media"}}

here we get a correct size ("flow_dst_tot_l4_payload_len":121587758) but we lose the 'hostname'. you could add the "hostname" when it's "end", it should be perfect...

bye Fred

[utoni]

The hostname needs to get stored into nDPId flow struct, because after nDPIs detection data was freed to save some heap memory, the hostname is gone. I'll fix that.

[UnveilTech]

Toni, excellent :) we'll be happy to test.

Fred

[utoni]

You may also use detected / detection-update events to save the hostname dissected by nDPI.

[UnveilTech]

we tried but we have no idea how to link infos "detected / detection-update" with "end/idle", any tips ? with "detected / detection-update" we find the hostname, with "end/idle" we find the correct size but no common ID or other to join those 2 infos.

Fred

[utoni]

You need to keep a state between a flow event new and an end / idle. In between those two events one detected and multiple detection-updates may occur. What's the language you want to implement this logic?

[UnveilTech]

we grap data from the log file that we compile to a SQL database. whatever the langage is, we need to use a common point to link the "new" to "end/idle" then find the hostname and the correct size. any idea ?

[utoni]

For that reasons, you may use the flow_id. In essence it's a numeric id which can be used to (uniquely) identify a flow and it's events.

[UnveilTech]

Toni,

we made several tests and we don't know if the bug is due to nDPId or the nDPI lib itself, we found many (too many) incorrect json lines in the log (nDPId). below is an extract of what we found: tmp.txt

2nd, we tried the flow_id, we could work with that items but it seems there is no (or so few) "end" or "idle" with DNS/DNS... protos, is that normal ?

anyway the best solution could be to have the hostname in the part "end"... ;o)

bye Fred

[utoni]

The file seems correct so far. There is no end / idle, because the DNS flows are still "active". Meaning there is data on the wire before a specific timeout hit's. That's the reason you get all those update events.

[UnveilTech]

Toni, verify lines from the file you'll see the json lines are not correct, lines seem corrupted or incompleted. website we use to check a json: https://www.functions-online.com/json_decode.html

Fred

[utoni]

Please note that nDPId is using it's own text protocol: https://github.com/utoni/nDPId?tab=readme-ov-file#json-stream-format

[UnveilTech]

we know that... we gave you the original lines, just try them with a json decoding (by removing the first 5 digits) and you'll see lines are corrupted/incompleted...

[utoni]

Now I am able to spot the issue. All lines in tmp.txt are malformed. What tool/application do you use to retrieve the events? Can you share some source code or shell commands how you did that?

[UnveilTech]

Toni,

we use PHP for reading the log file (easy for debugging). tool.php.txt (to rename .php)

you should find corrupted/malformed... :o)

Fred

[UnveilTech]

file tool.php.txt added to my previous answer, I was fighting with the github gui to upload the file...

[utoni]

The script seems to work. I've tried it with nc -U /tmp/ndpid-distributor.sock >/tmp/0.txt as well as with nc 127.0.0.1 7000 >/tmp/0.txt.

I've used echo $ayData['flow_id']."\n"; and echo $ayData."\n"; after $ayData = json_decode($szJson, true);. Both works for me.

Just getting a Warning if echoing $ayData: PHP Warning: Array to string conversion in /home/toni/Downloads/tool.php.txt on line 23

[UnveilTech]

if you get a warning Array/string, it means the flow_id struct is not correct (corrupted/malformed)...

please do the same with: ncat -U /tmp/ndpid-distributor.sock -l -k -o /tmp/0.txt

I cannot believe the issue is on our side only... ;o)

Fred

[utoni]

I get this warning for both statements. It doesn't matter if I am just printing the flow id from the JSON dictionary.

[UnveilTech]

Toni, we have tested with socat instead an unix socket and json lines are now correct. 200+ Mbps traffic and 1+ hour later we have not found any corrupted or malformed json, too weird... fyi: we use Debian 12 on Intel CPU.

so we keep with a socat for the moment, if you could at least fix the hostname it'll be perect :o)

have a nice sunday and take care, bye Fred

[utoni]

It's now possible to retrieve the hostname within analyse/end/idle flow events.

[UnveilTech]

Hi Toni, super ! we're going to test in a few minutes... keep you posted.

bye Fred

[UnveilTech]

it works like a charm, thanks a lot :)

bye Fred

[utoni]

config file support is work-in-progress :)

[UnveilTech]

man, you rock !!! :)