*** ndpi library version mismatch, expected API rev.10926 (4.9.0) found rev.11086 (4.9.0-4778-c2b657f) ***

[ronygut]

Using the latest versions of PF_RING and nDPI PF_RING 8.6.1 FROM October 23 nDPI 4.8 October 23

PF_RING FT is configured to use static nDPI ./configure --disable-ft-dl

When building PF_RING FT example with pcap and run ./ftflow_pcap -7 -i eth0 receiving: ndpi library version mismatch, expected API rev.10926 (4.9.0) found rev.11086 (4.9.0-4778-c2b657f)

Please advise. Thanks, Rony.

[cardigliano]

Please try pulling the latest code, we just updated the FT library to match latest nDPI

[ronygut]

It works now.

But now I see another problem. The sample program ftflow_pcap uses static ndpi library The sample program ftflow_pcap is crashing (with the following parameters -i ens160 -7) I also uncomment the line in the sample code : pfring_ft_set_l7_detected_callback(ft, l7Detected, NULL);

here are 2 crash back trace from GDB By the way it always crash with segmentation fault when I click on ctrl-c to stop the program.

Program received signal SIGSEGV, Segmentation fault. 0x00005555556b9281 in ndpi_serialize_binary_raw () (gdb) bt

0 0x00005555556b9281 in ndpi_serialize_binary_raw ()

1 0x00005555556c7112 in ndpi_flow2json ()

2 0x00005555556260c5 in processFlow ()

3 0x0000555555691ec0 in pfring_ft_housekeeping ()

4 0x0000555555692e4c in pfring_ft_process ()

5 0x000055555562551a in process_packet ()

6 0x000055555562a589 in pcap_handle_packet_mmap (handle=handle@entry=0x555555e0a8a0, callback=callback@entry=0x5555556254d0 , user=user@entry=0x0,

frame=frame@entry=0x7ffff72151d0 "\b\006", tp_len=<optimized out>, tp_mac=<optimized out>, tp_snaplen=1454, tp_sec=1722416485, tp_usec=48, tp_vlan_tci_valid=0, tp_vlan_tci=0, tp_vlan_tpid=33024)
at ./pcap-linux.c:4194

7 0x000055555562a994 in pcap_read_linux_mmap_v3 (handle=0x555555e0a8a0, max_packets=, callback=0x5555556254d0 , user=0x0) at ./pcap-linux.c:4344

and:

munmap_chunk(): invalid pointer

Program received signal SIGABRT, Aborted. __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350596416) at ./nptl/pthread_kill.c:44 44 ./nptl/pthread_kill.c: No such file or directory. (gdb) bt

0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350596416) at ./nptl/pthread_kill.c:44

1 __pthread_kill_internal (signo=6, threadid=140737350596416) at ./nptl/pthread_kill.c:78

2 __GI___pthread_kill (threadid=140737350596416, signo=signo@entry=6) at ./nptl/pthread_kill.c:89

3 0x00007ffff7ce4476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26

4 0x00007ffff7cca7f3 in __GI_abort () at ./stdlib/abort.c:79

5 0x00007ffff7d2b676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e7db8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155

6 0x00007ffff7d42cfc in malloc_printerr (str=str@entry=0x7ffff7e80230 "munmap_chunk(): invalid pointer") at ./malloc/malloc.c:5664

7 0x00007ffff7d42fdc in munmap_chunk (p=) at ./malloc/malloc.c:3060

8 0x00007ffff7d4749a in __GI___libc_free (mem=) at ./malloc/malloc.c:3381

9 0x00005555556b8fc9 in ndpi_realloc ()

10 0x00005555556bd2bd in ndpi_serialize_binary_uint32 ()

11 0x00005555556c727e in ndpi_flow2json ()

12 0x00005555556260c5 in processFlow ()

13 0x312e3237310c0070 in ?? ()

14 0xbb35322e31362e37 in ?? ()

15 0x695f747365640700 in ?? ()

16 0x32312e34370d0070 in ?? ()

17 0x3636312e38392e35 in ?? ()

18 0x0000000000000000 in ?? ()

[cardigliano]

@ronygut it seems I am not able to reproduce it, I will analyse the code, thank you for the trace

[cardigliano]

I was finally able to reproduce and fix this, please git pull the latest code and let me know

[ronygut]

Which git project to pull? nDPI , PF_RING, or both?

[cardigliano]

PF_RING

[ronygut]

Fixed! Thanks

[ronygut]

can I use pfring_ft_process from different threads? Is it thread safe? If I use PF_RING ZC with multiple consumer threads with pfring_zc_run_balancer_v2 can I use one flow table, or do I need to use flow table per consumer thread?

[cardigliano]

For performance reason FT does not implement locking and it has been designed to be used with one instance per thread in case of multithreading, rather than sharing a single table. In short, please allocate multiple tables, one per thread.

[ronygut]

in pfring_ft_create_table what does user_metadata_size is used for? If I set it to 0, how this influence the flow table behavior? v = pfring_ft_flow_get_value(flow); I use v->user in flow value to hold context for sessions from start of the session until the end of the session.

[ronygut]

v = pfring_ft_flow_get_value(flow); is v->user can be used to allocate application flow context? Is it the responsibility of the application to free any allocation pointed by v->user before calling pfring_ft_flow_free I use it to allocate an instance of a c++ class How do I use the v->user field to store application flow context?

[cardigliano]

in pfring_ft_create_table what does user_metadata_size is used for? If I set it to 0, how this influence the flow table behavior? v = pfring_ft_flow_get_value(flow); I use v->user in flow value to hold context for sessions from start of the session until the end of the session.

If you use v->user to store data, the size of that data should be provided via the user_metadata_size parameter, otherwise you access memory out of boundaries

[cardigliano]

v = pfring_ft_flow_get_value(flow); is v->user can be used to allocate application flow context? Is it the responsibility of the application to free any allocation pointed by v->user before calling pfring_ft_flow_free I use it to allocate an instance of a c++ class How do I use the v->user field to store application flow context?

FT takes care of allocating v->user (you can cast it to your data structure), just please pay attention if you place there pointers to other memory that you allocate dynamically (FT is not aware of those and you have to free them)

[ronygut]

pf_ring ft is responsible to allocate that size of bytes, and also to delete it at end of flow? The application just uses this buffer?

[cardigliano]

Correct

[ronygut]

Do the callback from pfring_ft_set_flow_packet_callback is always called for each packet added to the flow? when l7 classification callback is called is it the same packet that will be called on the callback from pfring_ft_set_flow_packet_callback? Which callback from the two will be called first? the l7 callback is not called if the flow is short and it was not possible to do DPI? In this case the flow protocol is set to Unknown, or l7 callback should always be called even on short flows?

[cardigliano]

Do the callback from pfring_ft_set_flow_packet_callback is always called for each packet added to the flow? Yes when l7 classification callback is called is it the same packet that will be called on the callback from pfring_ft_set_flow_packet_callback? L7 classification is called one per flow, as soon as the application is detected Which callback from the two will be called first? flow_packet_callback after the l7 callback is not called if the flow is short and it was not possible to do DPI? No In this case the flow protocol is set to Unknown, or l7 callback should always be called even on short flows? l7 callback is not called in that case, you should use the export callback to check the protocol at flow termination

[ronygut]

is HBO - host order? source and destination is changed in each packet in the flow. When do you determine the client and server setting for IPs and ports? e.g. client is the initiating of the flow in TCP.

In pfring_ft_flow_key , the saddr ,sport and daddr, dport are the client and server for the flow?

The direction in metadata is changed to s2d_direction and d2s_direction according to the packet itself?

The saddr ,sport and daddr, dport in pfring_ft_flow_key will always be the same in the flow? are they indicate client/server?

[cardigliano]

is HBO - host order? Yes, Host Byte Order

source and destination is changed in each packet in the flow. When do you determine the client and server setting for IPs and ports? e.g. client is the initiating of the flow in TCP. Client is the src IP from the first packet of the flow

In pfring_ft_flow_key , the saddr ,sport and daddr, dport are the client and server for the flow? Yes

The direction in metadata is changed to s2d_direction and d2s_direction according to the packet itself? Yes

The saddr ,sport and daddr, dport in pfring_ft_flow_key will always be the same in the flow? are they indicate client/server? Yes

[ronygut]

Do I need to call pfring_ft_housekeeping in the loop that process packets every x seconds for each flow table I use? I currently do not call it and if I remember correctly I saw that it was called by the Flow Table implicitly? (saw it in the back trace of the debugger)

[ronygut]

Does VLAN ID part of the flow key by default? I know it is part of the key structure, but does the flow key consists of vlan id + src ip + src port + dst ip + dst port + protocol? If yes, is it possible to configure that vlan id will not be part of the unique flow key?

I use only IPV4

I'm creating 12 threads and each has a reference to different pfring_ft_table All threads are using the same callbacks (start flow, end flow etc...) routines that will be called from different threads. The callbacks know to which flow table they are using as they have the right reference in the user pointer of the callback. (flow table index) I can see that I get the same flow with different vlan ID (0 and 200) called on the new flow callback for the same thread. The strange thing is that if I use TCPDUMP to see packets with VLAN ID 0 I don't see such packets at all. tcpdump -i ens160 -nn vlan 0 and ip -e I can only see packets with VLAN ID 200 tcpdump -i ens160 -nn vlan 200 and ip -e How come PF_RING FT see such flows with VLAN ID 0? By the way I can see VLAN ID 0 also in the example_ft program ftflow. vlanId: 0, srcIp: 10.168.233.137, dstIp: 10.201.55.103, srcPort: 43832, dstPort: 9093

Maybe, VLAN ID 0 indicates that there was no VLAN in the packet? If so, how do you distinguish between no VLAN ID and VLAN ID 0 in the packet?

[ronygut]

What does pfring_ft_ext_pkthdr is used for in pfring_ft_create_table. I see that in the ftflow.c you pass the packet hash

[cardigliano]

Do I need to call pfring_ft_housekeeping in the loop that process packets every x seconds for each flow table I use? I currently do not call it and if I remember correctly I saw that it was called by the Flow Table implicitly? (saw it in the back trace of the debugger)

pfring_ft_housekeeping is called automatically once per second when calling pfring_ft_process(), however it is a good practice to call it by the caller when idle (in case pfring_ft_process is not called). However it is not mandatory to call it.

[cardigliano]

What does pfring_ft_ext_pkthdr is used for in pfring_ft_create_table. I see that in the ftflow.c you pass the packet hash

This is not actually used by FT itseld, it is for external use.

[cardigliano]

Does VLAN ID part of the flow key by default? I know it is part of the key structure, but does the flow key consists of vlan id + src ip + src port + dst ip + dst port + protocol? If yes, is it possible to configure that vlan id will not be part of the unique flow key?

Yes, vlan id is part of the flow key. We can make it optional (open a separate ticket with this feature request)

I use only IPV4

I'm creating 12 threads and each has a reference to different pfring_ft_table All threads are using the same callbacks (start flow, end flow etc...) routines that will be called from different threads. The callbacks know to which flow table they are using as they have the right reference in the user pointer of the callback. (flow table index) I can see that I get the same flow with different vlan ID (0 and 200) called on the new flow callback for the same thread. The strange thing is that if I use TCPDUMP to see packets with VLAN ID 0 I don't see such packets at all. tcpdump -i ens160 -nn vlan 0 and ip -e I can only see packets with VLAN ID 200 tcpdump -i ens160 -nn vlan 200 and ip -e How come PF_RING FT see such flows with VLAN ID 0? By the way I can see VLAN ID 0 also in the example_ft program ftflow. vlanId: 0, srcIp: 10.168.233.137, dstIp: 10.201.55.103, srcPort: 43832, dstPort: 9093

Could you provide a pcap to reproduce this?

Maybe, VLAN ID 0 indicates that there was no VLAN in the packet? If so, how do you distinguish between no VLAN ID and VLAN ID 0 in the packet?

Yes, 0 means no vlan for us. Do you have a vlan with id 0?

[ronygut]

Thank you for your answers, I really appreciate it! Another important question. I want to use pf_ring ft in multithreaded environment. I use PF_RINF ZC to create multiple consumer threads depending on configuration and expected load of the environment. I use the default Built in IP based hash to distribute the load across threads. Each threads is bind to a core. Each thread gets its own traffic and handle it. For performance , I want to use PF_RING FT (create Flow Table) for each thread. How do you recommend creating each flow table that will be used in each consumer thread? Do I need to declare __thread pfring_ft_tableto* ft to use by pfring_ft_create_table? in each consumer thread?( I mean to use thread local storage declaration?)

Can I use an array of pfring_ft_tableto* , according to the number of consumer threads that I create, dynamically, and each thread will have an index to this array? (I will call pfring_ft_create_table in a main thread and fill it up before consumer threads are created and each can use one table according to an index in the array)

What is the recommended way? I want to avoid synchronizations between PF_RING ZC consumer threads on the flow table? Also all callback functions that I declare , can they be shared across threads by using the index that is dedicated to a specific consumer thread?

If you have an example of using multiple pf_ring flow table in multiple threads I would appreciate that you can share the code as an example on how to correctly do it.

[ronygut]

Does VLAN ID part of the flow key by default? I know it is part of the key structure, but does the flow key consists of vlan id + src ip + src port + dst ip + dst port + protocol? If yes, is it possible to configure that vlan id will not be part of the unique flow key?

Yes, vlan id is part of the flow key. We can make it optional (open a separate ticket with this feature request)

I use only IPV4 I'm creating 12 threads and each has a reference to different pfring_ft_table All threads are using the same callbacks (start flow, end flow etc...) routines that will be called from different threads. The callbacks know to which flow table they are using as they have the right reference in the user pointer of the callback. (flow table index) I can see that I get the same flow with different vlan ID (0 and 200) called on the new flow callback for the same thread. The strange thing is that if I use TCPDUMP to see packets with VLAN ID 0 I don't see such packets at all. tcpdump -i ens160 -nn vlan 0 and ip -e I can only see packets with VLAN ID 200 tcpdump -i ens160 -nn vlan 200 and ip -e How come PF_RING FT see such flows with VLAN ID 0? By the way I can see VLAN ID 0 also in the example_ft program ftflow. vlanId: 0, srcIp: 10.168.233.137, dstIp: 10.201.55.103, srcPort: 43832, dstPort: 9093

Could you provide a pcap to reproduce this?

Maybe, VLAN ID 0 indicates that there was no VLAN in the packet? If so, how do you distinguish between no VLAN ID and VLAN ID 0 in the packet?

Yes, 0 means no vlan for us. Do you have a vlan with id 0?

No we don't have VLAN ID 0. I understand that in this case it is populate with default 0 in the VLAN ID

[ronygut]

Where can I open the ticket for the VLAN ID field that will be an optional in the flow key?

[cardigliano]

Sometimes when I call pfring_ft_get_stats I get on err_no_mem > 0 for example err_no_mem: 8 err_no_mem: 4 What does it mean? What do I need to do in order to fix it? err_no_room is always 0 stats: 2024-08-23 13:54:41:004 [140208568772160(2083878)] {PfRingZc.cpp:964} MESSAGE Per thread[8] => Active Flows : 1734 Flows: 3244 0 err_no_room: 0 err_no_mem: 8 2024-08-23 13:54:41:005 [140209687590464(2083866)] {PfRingZc.cpp:964} MESSAGE Per thread[4] => Active Flows : 922 Flows: 2052 0 err_no_room: 0 err_no_mem: 4 2024-08-23 13:54:41:005 [140210063005248(2083860)] {PfRingZc.cpp:964} MESSAGE Per thread[2] => Active Flows : 894 Flows: 1711 0 err_no_room: 0 err_no_mem: 2 2024-08-23 13:54:41:005 [140208992409152(2083872)] {PfRingZc.cpp:964} MESSAGE Per thread[6] => Active Flows : 1082 Flows: 2090 0 err_no_room: 0 err_no_mem: 6 2024-08-23 13:54:41:005 [140209374078528(2083869)] {PfRingZc.cpp:964} MESSAGE Per thread[5] => Active Flows : 754 Flows: 1504 0 err_no_room: 0 err_no_mem: 5 2024-08-23 13:54:41:006 [140209735853632(2083863)] {PfRingZc.cpp:964} MESSAGE Per thread[3] => Active Flows : 1354 Flows: 2673 0 err_no_room: 0 err_no_mem: 3 2024-08-23 13:54:41:007 [140210541160000(2083853)] {PfRingZc.cpp:964} MESSAGE Per thread[0] => Active Flows : 950 Flows: 2111 0 err_no_room: 0 err_no_mem: 0 2024-08-23 13:54:41:009 [140208949392960(2083875)] {PfRingZc.cpp:964} MESSAGE Per thread[7] => Active Flows : 1250 Flows: 2513 0 err_no_room: 0 err_no_mem: 7

This means the number of active flows is exceeding the flow table size. You need to increase the max_flows parameters when creating the table with pfring_ft_create_table

[cardigliano]

Where can I open the ticket for the VLAN ID field that will be an optional in the flow key?

Here on github

[ronygut]

In the struct pfring_ft_flow_dir_value the field bytes is the size in bytes of data bytes per direction, or total bytes (including headers) per direction?

[cardigliano]

In the struct pfring_ft_flow_dir_value the field bytes is the size in bytes of data bytes per direction, or total bytes (including headers) per direction?

It's the total bytes per direction

[ronygut]

when using pfring_ft_flow_set_flow_slicing for example with 300 seconds. The callback set with pfring_ft_set_flow_export_callback will be called every 300 seconds when the flow is still active? What are all scenarios that the export callback will be called related to slicing timeout?

For some reason I see that the export callback for slicing is called much less then the timeout was set. For the first time is called after 300 seconds, but the next callbacks after that, are called for much less time and a lot of times for specific flow. (even every second) What can be the reason for that?

[cardigliano]

when using pfring_ft_flow_set_flow_slicing for example with 300 seconds. The callback set with pfring_ft_set_flow_export_callback will be called every 300 seconds when the flow is still active?

Correct

What are all scenarios that the export callback will be called related to slicing timeout?

For some reason I see that the export callback for slicing is called much less then the timeout was set. For the first time is called after 300 seconds, but the next callbacks after that, are called for much less time and a lot of times for specific flow. (even every second) What can be the reason for that?

Maybe the flow terminates for other reason? (e.g. tcp termination, or configured timeouts)

[ronygut]

In case the flow terminates I should get the status of termination (PFRING_FT_FLOW_STATUS_END_DETECTED, or PFRING_FT_FLOW_STATUS_IDLE_TIMEOUT) in the callback from status field and not PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT, right?

[cardigliano]

Correct. I will double check and run some test.

[ronygut]

Another questions regarding memory leaks. I used valgrind to check for memory leaks and this is the report I see:

What I'm concern is the leaks it reports regarding pfring_ft_flow_slice. I run the housekeeping function every ~ 5 seconds. Your advise is appreciated.

==2189580== HEAP SUMMARY: ==2189580== in use at exit: 305,808,248 bytes in 200,770 blocks ==2189580== total heap usage: 8,624,245 allocs, 8,423,475 frees, 1,608,996,829 bytes allocated ==2189580== ==2189580== Thread 1: ==2189580== 8 bytes in 1 blocks are definitely lost in loss record 226 of 1,457 ==2189580== at 0x4849013: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x210E6C: main (in /home/emperor/ndr) ==2189580== ==2189580== 312 bytes in 1 blocks are definitely lost in loss record 1,002 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F121B: pfring_ft_hash_check_expired_flows (in /home/emperor/ndr) ==2189580== by 0x2E17C5: pfring_ft_housekeeping (in /home/emperor/ndr) ==2189580== by 0x22FF96: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== 384 bytes in 1 blocks are possibly lost in loss record 1,036 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x40147D9: calloc (rtld-malloc.h:44) ==2189580== by 0x40147D9: allocate_dtv (dl-tls.c:375) ==2189580== by 0x40147D9: _dl_allocate_tls (dl-tls.c:634) ==2189580== by 0x51357B4: allocate_stack (allocatestack.c:430) ==2189580== by 0x51357B4: pthread_create@@GLIBC_2.34 (pthread_create.c:647) ==2189580== by 0x2B3E14: pfring_zc_run_balancer_v2 (in /home/emperor/ndr) ==2189580== by 0x23B0FF: PfRingZc::StartBalancer(void ()(char, PfRingZc), PfRingZc) (in /home/emperor/ndr) ==2189580== by 0x23CEA9: PfRingZc::PfRingZc(char, Configuration&, PCF_Event) (in /home/emperor/ndr) ==2189580== by 0x2110ED: main (in /home/emperor/ndr) ==2189580== ==2189580== 384 bytes in 1 blocks are possibly lost in loss record 1,037 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x40147D9: calloc (rtld-malloc.h:44) ==2189580== by 0x40147D9: allocate_dtv (dl-tls.c:375) ==2189580== by 0x40147D9: _dl_allocate_tls (dl-tls.c:634) ==2189580== by 0x51357B4: allocate_stack (allocatestack.c:430) ==2189580== by 0x51357B4: pthread_create@@GLIBC_2.34 (pthread_create.c:647) ==2189580== by 0x5140C04: timer_start_helper_thread (timer_routines.c:147) ==2189580== by 0x5139EE7: pthread_once_slow (pthread_once.c:116) ==2189580== by 0x514060A: timer_create@@GLIBC_2.34 (timer_create.c:70) ==2189580== by 0x23DC45: PCF_Timer::PCF_Timer(int, void ()(void), void, bool) (in /home/emperor/ndr) ==2189580== by 0x23B15D: PfRingZc::StartBalancer(void ()(char, PfRingZc), PfRingZc) (in /home/emperor/ndr) ==2189580== by 0x23CEA9: PfRingZc::PfRingZc(char, Configuration&, PCF_Event) (in /home/emperor/ndr) ==2189580== by 0x2110ED: main (in /home/emperor/ndr) ==2189580== ==2189580== 384 bytes in 1 blocks are possibly lost in loss record 1,038 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x40147D9: calloc (rtld-malloc.h:44) ==2189580== by 0x40147D9: allocate_dtv (dl-tls.c:375) ==2189580== by 0x40147D9: _dl_allocate_tls (dl-tls.c:634) ==2189580== by 0x51357B4: allocate_stack (allocatestack.c:430) ==2189580== by 0x51357B4: pthread_create@@GLIBC_2.34 (pthread_create.c:647) ==2189580== by 0x2145CB: PCF_Thread::create_thread(void ()(void), void) (in /home/emperor/ndr) ==2189580== by 0x23B1D1: PfRingZc::StartBalancer(void ()(char, PfRingZc), PfRingZc) (in /home/emperor/ndr) ==2189580== by 0x23CEA9: PfRingZc::PfRingZc(char, Configuration&, PCF_Event) (in /home/emperor/ndr) ==2189580== by 0x2110ED: main (in /home/emperor/ndr) ==2189580== ==2189580== 384 bytes in 1 blocks are possibly lost in loss record 1,039 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x40147D9: calloc (rtld-malloc.h:44) ==2189580== by 0x40147D9: allocate_dtv (dl-tls.c:375) ==2189580== by 0x40147D9: _dl_allocate_tls (dl-tls.c:634) ==2189580== by 0x51357B4: allocate_stack (allocatestack.c:430) ==2189580== by 0x51357B4: pthread_create@@GLIBC_2.34 (pthread_create.c:647) ==2189580== by 0x2145CB: PCF_Thread::create_thread(void ()(void), void) (in /home/emperor/ndr) ==2189580== by 0x23B337: PfRingZc::StartBalancer(void ()(char, PfRingZc), PfRingZc) (in /home/emperor/ndr) ==2189580== by 0x23CEA9: PfRingZc::PfRingZc(char, Configuration&, PCF_Event) (in /home/emperor/ndr) ==2189580== by 0x2110ED: main (in /home/emperor/ndr) ==2189580== ==2189580== 624 bytes in 2 blocks are possibly lost in loss record 1,097 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2E230F: pfring_ft_process (in /home/emperor/ndr) ==2189580== by 0x22FF6B: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== 656 bytes in 1 blocks are definitely lost in loss record 1,103 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2AC499: pfring_mod_zc_dev_open (in /home/emperor/ndr) ==2189580== by 0x2AAF14: pfring_zc_open (in /home/emperor/ndr) ==2189580== by 0x29C6B2: pfring_open (in /home/emperor/ndr) ==2189580== by 0x23A3DC: PfRingZc::OpenDevices() (in /home/emperor/ndr) ==2189580== by 0x23CE8F: PfRingZc::PfRingZc(char, Configuration&, PCF_Event) (in /home/emperor/ndr) ==2189580== by 0x2110ED: main (in /home/emperor/ndr) ==2189580== ==2189580== 2,344 (936 direct, 1,408 indirect) bytes in 3 blocks are definitely lost in loss record 1,227 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F1120: pfring_ft_hash_harvest_expired_flows (in /home/emperor/ndr) ==2189580== by 0x2E17F0: pfring_ft_housekeeping (in /home/emperor/ndr) ==2189580== by 0x22FF96: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== 3,432 bytes in 11 blocks are possibly lost in loss record 1,254 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F1120: pfring_ft_hash_harvest_expired_flows (in /home/emperor/ndr) ==2189580== by 0x2E17F0: pfring_ft_housekeeping (in /home/emperor/ndr) ==2189580== by 0x2E285B: pfring_ft_process (in /home/emperor/ndr) ==2189580== by 0x22FF6B: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== 11,761 (4,992 direct, 6,769 indirect) bytes in 16 blocks are definitely lost in loss record 1,306 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F121B: pfring_ft_hash_check_expired_flows (in /home/emperor/ndr) ==2189580== by 0x2E17C5: pfring_ft_housekeeping (in /home/emperor/ndr) ==2189580== by 0x2E285B: pfring_ft_process (in /home/emperor/ndr) ==2189580== by 0x22FF6B: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== 92,388 (28,704 direct, 63,684 indirect) bytes in 92 blocks are definitely lost in loss record 1,393 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2E230F: pfring_ft_process (in /home/emperor/ndr) ==2189580== by 0x22FF6B: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== 624,684 (212,784 direct, 411,900 indirect) bytes in 682 blocks are definitely lost in loss record 1,428 of 1,457 ==2189580== at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so) ==2189580== by 0x2D7A55: pfring_ft_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F0C67: pfring_ft_hash_export_flow_slice (in /home/emperor/ndr) ==2189580== by 0x2F1120: pfring_ft_hash_harvest_expired_flows (in /home/emperor/ndr) ==2189580== by 0x2E17F0: pfring_ft_housekeeping (in /home/emperor/ndr) ==2189580== by 0x2E285B: pfring_ft_process (in /home/emperor/ndr) ==2189580== by 0x22FF6B: FlowManager::pfringzc_process_packet(void, pcap_pkthdr const, unsigned char const, long) (in /home/emperor/ndr) ==2189580== by 0x23AF24: PfRingZc::ConsumerThread(void) (in /home/emperor/ndr) ==2189580== by 0x21360E: PCF_Thread::InternalThread(void) (in /home/emperor/ndr) ==2189580== by 0x5134AC2: start_thread (pthread_create.c:442) ==2189580== by 0x51C5BF3: clone (clone.S:100) ==2189580== ==2189580== LEAK SUMMARY: ==2189580== definitely lost: 248,392 bytes in 796 blocks ==2189580== indirectly lost: 483,761 bytes in 1,320 blocks ==2189580== possibly lost: 5,592 bytes in 17 blocks ==2189580== still reachable: 305,070,503 bytes in 198,637 blocks ==2189580== suppressed: 0 bytes in 0 blocks ==2189580== Reachable blocks (those to which a pointer was found) are not shown. ==2189580== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==2189580==

[cardigliano]

@ronygut I am checking the slicing code, I suggest to use just flow timeouts to control the export until I review the slicing support (this was implemented for a customer time ago, I do not think others are using this)

[ronygut]

@cardigliano , we also need the flow slicing feature. We need to get indication every x seconds for long term flows in order to create some information every x seconds. Please try to see why it is leaking memory and also please check that it calls the export callback at the correct intervals that was set with the pfring_ft_flow_set_flow_slicing function.

[cardigliano]

@ronygut I pushed a fix for the flow slices duration. I will check the leaks

[ronygut]

@cardigliano , I just took the latest PF_RING 8.8.0 and nDPI 4.10 from git and I get the mismatch version error: ndpi library version mismatch, expected API rev.11192 (4.11.0) found rev.4108 (4.11.0) Can you please fix it?

[cardigliano]

Please pull the latest code from the 8.8 branch

[cardigliano]

I am not able to reproduce the "pfring_ft_flow_slice" leaks using ftflow, I see you are using your application. Could you try with ftflow? And could you provide instructions in that case? Thank you.

[cardigliano]

Are you calling pfring_ft_flow_free for flows exported to the callback defined with pfring_ft_set_flow_export_callback?

[ronygut]

@cardigliano I use the zip file from the GIT repository. Did you update the zip file in PF_RING as well? I downloaded the zip file and extract it: then cd userland and run ./configure --disable-ft-dl then run make Build everything with new libpfring.a

and I still get: ndpi library version mismatch, expected API rev.11192 (4.11.0) found rev.4108 (4.11.0)

[ronygut]

Regarding the memory leak, please make sure to use also:

pfring_ft_set_flow_packet_callback pfring_ft_set_new_flow_callback and pfring_ft_set_l7_detected_callback

I use in my code also those callbacks

[cardigliano]

@cardigliano I use the zip file from the GIT repository. Did you update the zip file in PF_RING as well? I downloaded the zip file and extract it: then cd userland and run ./configure --disable-ft-dl then run make Build everything with new libpfring.a

and I still get: ndpi library version mismatch, expected API rev.11192 (4.11.0) found rev.4108 (4.11.0)

I did not update the zip, we need to release a new minor release for that

[cardigliano]

Regarding the memory leak, please make sure to use also:

pfring_ft_set_flow_packet_callback pfring_ft_set_new_flow_callback and pfring_ft_set_l7_detected_callback

I use in my code also those callbacks

I tried enabling all the callbacks, still no leak. Please provide a small pcap with traffic able to reproduce this using ftflow

[ronygut]

@cardigliano I think I know what is the main difference between my program and ftflow.c After enabling the slicing, In the callback of export flow (processFlow) I check for the status and if it is PFRING_FT_FLOW_STATUS_SLICE_TIMEOUT I don't free the flow as the flow is not ended yet. (e.g. not calling pfring_ft_flow_free(flow);) In the ftflow.c on every call to processFlow callback it calls pfring_ft_flow_free to free the flow. However in case of slicing you should not free the flow and continue. Did you free the flow also on slicing?

[cardigliano]

Please free the flow, regardless of the status. Slices are is just a copy of the original flow.