search

ntop / nDPI

Open Source Deep Packet Inspection Software Toolkit
http://www.ntop.org
GNU Lesser General Public License v3.0
3.8k stars 894 forks source link

Different Youtube classification on little-endian x big-endian machine #1475

Closed viniciussn closed 2 years ago

viniciussn commented 2 years ago

Hello,

Running ndpiReader with a Youtube traffic pcap is giving me different outputs on a little-endian and a big-endian machine with the latest commit (95a3d4fffe699823b1cc9730dd0e4e2827b94845).

Little-endian output ``` root@machine# ./ndpiReader -i youtube.pcap ----------------------------------------------------------- * NOTE: This is demo app to show *some* nDPI features. * In this demo we have implemented only some basic features * just to show you what you can do with the library. Feel * free to extend it and send us the patches for inclusion ------------------------------------------------------------ Using nDPI (4.3.0-3520-95a3d4ff) [1 thread(s)] Using libgcrypt version 1.8.6internal Reading packets from pcap file /home/vinicius/Downloads/subir/youtube.pcap... Running thread 0... nDPI Memory statistics: nDPI Memory (once): 237.23 KB Flow Memory (per flow): 688 B Actual Memory: 25.00 MB Peak Memory: 25.00 MB Setup Time: 37 msec Packet Processing Time: 18 msec Traffic statistics: Ethernet bytes: 21308413 (includes ethernet CRC/IFC/trailer) Discarded bytes: 0 IP packets: 19999 of 19999 packets total IP bytes: 20828437 (avg pkt size 1041 bytes) Unique flows: 47 TCP Packets: 2857 UDP Packets: 17140 VLAN Packets: 0 MPLS Packets: 0 PPPoE Packets: 0 Fragmented Packets: 0 Max Packet size: 5704 Packet Len < 64: 5611 Packet Len 64-128: 151 Packet Len 128-256: 59 Packet Len 256-1024: 199 Packet Len 1024-1500: 13562 Packet Len > 1500: 417 nDPI throughput: 1.09 M pps / 8.64 Gb/sec Analysis begin: 04/Mar/2022 11:22:52 Analysis end: 04/Mar/2022 11:23:22 Traffic throughput: 657.37 pps / 5.34 Mb/sec Traffic duration: 30.423 sec Guessed flow protos: 10 DPI Packets (TCP): 249 (8.03 pkts/flow) DPI Packets (UDP): 16 (1.07 pkts/flow) DPI Packets (other): 1 (1.00 pkts/flow) Confidence: Match by port 6 (flows) Confidence: Match by IP 4 (flows) Confidence: DPI 37 (flows) Detected protocols: DNS packets: 2 bytes: 190 flows: 1 HTTP packets: 20 bytes: 1320 flows: 3 OCSP packets: 102 bytes: 30637 flows: 6 ICMP packets: 2 bytes: 196 flows: 1 SSH packets: 70 bytes: 9884 flows: 2 YouTube packets: 19098 bytes: 20454926 flows: 14 Google packets: 585 bytes: 298742 flows: 13 TeamViewer packets: 5 bytes: 348 flows: 1 Cloudflare packets: 10 bytes: 570 flows: 1 GoogleServices packets: 87 bytes: 30436 flows: 2 AmazonAWS packets: 6 bytes: 396 flows: 1 GoogleCloud packets: 12 bytes: 792 flows: 2 Protocol statistics: Safe 30637 bytes Acceptable 342874 bytes Fun 20454926 bytes ```
Big-endian output ``` root@machine# ./ndpiReader -i youtube.pcap ----------------------------------------------------------- * NOTE: This is demo app to show *some* nDPI features. * In this demo we have implemented only some basic features * just to show you what you can do with the library. Feel * free to extend it and send us the patches for inclusion ------------------------------------------------------------ Using nDPI (4.3.0-3520-95a3d4ff) [1 thread(s)] Using libgcrypt version 1.8.6internal Reading packets from pcap file youtube.pcap... Running thread 0... nDPI Memory statistics: nDPI Memory (once): 223.71 KB Flow Memory (per flow): 632 B Actual Memory: 23.26 MB Peak Memory: 23.26 MB Setup Time: 769 msec Packet Processing Time: 729 msec Traffic statistics: Ethernet bytes: 21308413 (includes ethernet CRC/IFC/trailer) Discarded bytes: 0 IP packets: 19999 of 19999 packets total IP bytes: 20828437 (avg pkt size 1041 bytes) Unique flows: 47 TCP Packets: 2857 UDP Packets: 17140 VLAN Packets: 0 MPLS Packets: 0 PPPoE Packets: 0 Fragmented Packets: 0 Max Packet size: 5704 Packet Len < 64: 5611 Packet Len 64-128: 151 Packet Len 128-256: 59 Packet Len 256-1024: 199 Packet Len 1024-1500: 13562 Packet Len > 1500: 417 nDPI throughput: 27.42 K pps / 222.87 Mb/sec Analysis begin: 04/Mar/2022 14:22:52 Analysis end: 04/Mar/2022 14:23:22 Traffic throughput: 657.37 pps / 5.34 Mb/sec Traffic duration: 30.423 sec Guessed flow protos: 10 DPI Packets (TCP): 249 (8.03 pkts/flow) DPI Packets (UDP): 16 (1.07 pkts/flow) DPI Packets (other): 1 (1.00 pkts/flow) Confidence: Match by port 6 (flows) Confidence: Match by IP 4 (flows) Confidence: DPI 37 (flows) Detected protocols: DNS packets: 2 bytes: 190 flows: 1 HTTP packets: 20 bytes: 1320 flows: 3 OCSP packets: 102 bytes: 30637 flows: 6 ICMP packets: 2 bytes: 196 flows: 1 SSH packets: 70 bytes: 9884 flows: 2 YouTube packets: 2247 bytes: 2605867 flows: 8 Google packets: 1972 bytes: 1561744 flows: 18 TeamViewer packets: 5 bytes: 348 flows: 1 QUIC packets: 15510 bytes: 16605433 flows: 2 Cloudflare packets: 10 bytes: 570 flows: 1 GoogleServices packets: 41 bytes: 11060 flows: 1 AmazonAWS packets: 6 bytes: 396 flows: 1 GoogleCloud packets: 12 bytes: 792 flows: 2 Protocol statistics: Safe 30637 bytes Acceptable 18191933 bytes Fun 2605867 bytes ```

Command used: ndpiReader -i youtube.pcap

Example: youtube.pcap

utoni commented 2 years ago

Possible duplicate of #1312.

viniciussn commented 2 years ago

Do you agree with @IvanNardi?

these issues seem to be related to ndpiReader application, not libndpi.so itself

Because I'm using libndpi in another application and this issue is also reproducible there.

IvanNardi commented 2 years ago

Do you agree with @IvanNardi?

these issues seem to be related to ndpiReader application, not libndpi.so itself

Because I'm using libndpi in another application and this issue is also reproducible there.

Please note that comment has been made before the addition of the internal crypto code. I would not be surprise at all if that code has some issues on big-endian machines.

IvanNardi commented 2 years ago

Definitely some endianess issue on internal crypto library: https://github.com/ntop/nDPI/runs/5424718108?check_suite_focus=true all QUIC tests are failing on big-endian arch

vel21ripn commented 2 years ago

Does the error only appear when internal gcrypt is used? At the time of the adoption of #1444 there was no testing on s390 with built-in libgcrypt?

viniciussn commented 2 years ago

I'm building the latest version with host libgcrypt and will post the results ASAP.

viniciussn commented 2 years ago

Here is the output of ndpiReader with libgcrypt 1.9.3, running on a big-endian machine:

output ``` root@machine# ./ndpiReader -i youtube.pcap ----------------------------------------------------------- * NOTE: This is demo app to show *some* nDPI features. * In this demo we have implemented only some basic features * just to show you what you can do with the library. Feel * free to extend it and send us the patches for inclusion ------------------------------------------------------------ Using nDPI (4.3.0-3520-95a3d4ff) [1 thread(s)] Using libgcrypt version 1.9.3 Reading packets from pcap file youtube.pcap... Running thread 0... nDPI Memory statistics: nDPI Memory (once): 223.71 KB Flow Memory (per flow): 632 B Actual Memory: 23.21 MB Peak Memory: 23.21 MB Setup Time: 796 msec Packet Processing Time: 784 msec Traffic statistics: Ethernet bytes: 21308413 (includes ethernet CRC/IFC/trailer) Discarded bytes: 0 IP packets: 19999 of 19999 packets total IP bytes: 20828437 (avg pkt size 1041 bytes) Unique flows: 47 TCP Packets: 2857 UDP Packets: 17140 VLAN Packets: 0 MPLS Packets: 0 PPPoE Packets: 0 Fragmented Packets: 0 Max Packet size: 5704 Packet Len < 64: 5611 Packet Len 64-128: 151 Packet Len 128-256: 59 Packet Len 256-1024: 199 Packet Len 1024-1500: 13562 Packet Len > 1500: 417 nDPI throughput: 25.50 K pps / 207.26 Mb/sec Analysis begin: 04/Mar/2022 14:22:52 Analysis end: 04/Mar/2022 14:23:22 Traffic throughput: 657.37 pps / 5.34 Mb/sec Traffic duration: 30.423 sec Guessed flow protos: 10 DPI Packets (TCP): 249 (8.03 pkts/flow) DPI Packets (UDP): 16 (1.07 pkts/flow) DPI Packets (other): 1 (1.00 pkts/flow) Confidence: Match by port 6 (flows) Confidence: Match by IP 4 (flows) Confidence: DPI 37 (flows) Detected protocols: DNS packets: 2 bytes: 190 flows: 1 HTTP packets: 20 bytes: 1320 flows: 3 OCSP packets: 102 bytes: 30637 flows: 6 ICMP packets: 2 bytes: 196 flows: 1 SSH packets: 70 bytes: 9884 flows: 2 YouTube packets: 19098 bytes: 20454926 flows: 14 Google packets: 585 bytes: 298742 flows: 13 TeamViewer packets: 5 bytes: 348 flows: 1 Cloudflare packets: 10 bytes: 570 flows: 1 GoogleServices packets: 87 bytes: 30436 flows: 2 AmazonAWS packets: 6 bytes: 396 flows: 1 GoogleCloud packets: 12 bytes: 792 flows: 2 Protocol statistics: Safe 30637 bytes Acceptable 342874 bytes Fun 20454926 bytes ```
IvanNardi commented 2 years ago

I confirm: QUIC dissection is fine with external libgcrypt on BE machines: https://github.com/ntop/nDPI/runs/5426850761?check_suite_focus=true

vel21ripn commented 2 years ago

We need to check the functionality of the "tests/performance/gcrypt" utilities on the s390. I made a test system on s390/qemu/ubuntu-18. I'll try to figure out where the error is.

vel21ripn commented 2 years ago

See #1478 I see strange test errors on s390 (not related to Endian issues). 1kxun.pcap: Number of packets and bytes do not match. ethernetIP.pcap: category does not match. KakaoTalk_chat.pcap: errors related to the ICMP protocol. quic_interop_V.pcapng: errors related to the ICMP protocol. skype_no_unknown.pcap,teams.pcap,zoom.pcap: The number of packets and bytes does not match.

IvanNardi commented 2 years ago

EthernetIP should be fixed in #1477 @lnslbrty , it seems that ICMP checksum calculation (or comparison) is wrong on BE machines. Could you take a look, please?

The other issues (already mentioned in https://github.com/ntop/nDPI/issues/1312#issuecomment-1024892771) seems to be related to ndpiReader code.

@viniciussn , could you confirm that this specific issue can be close, please?

viniciussn commented 2 years ago

Hello,

Now it is working as expected. Output on a big-endian machine:

output ``` root@machine# ./ndpiReader -i youtube.pcap ----------------------------------------------------------- * NOTE: This is demo app to show *some* nDPI features. * In this demo we have implemented only some basic features * just to show you what you can do with the library. Feel * free to extend it and send us the patches for inclusion ------------------------------------------------------------ Using nDPI (4.3.0-3523-c345b3c7) [1 thread(s)] Using libgcrypt version 1.8.6internal Reading packets from pcap file youtube.pcap... Running thread 0... nDPI Memory statistics: nDPI Memory (once): 223.71 KB Flow Memory (per flow): 632 B Actual Memory: 23.26 MB Peak Memory: 23.26 MB Setup Time: 753 msec Packet Processing Time: 737 msec Traffic statistics: Ethernet bytes: 21308413 (includes ethernet CRC/IFC/trailer) Discarded bytes: 0 IP packets: 19999 of 19999 packets total IP bytes: 20828437 (avg pkt size 1041 bytes) Unique flows: 47 TCP Packets: 2857 UDP Packets: 17140 VLAN Packets: 0 MPLS Packets: 0 PPPoE Packets: 0 Fragmented Packets: 0 Max Packet size: 5704 Packet Len < 64: 5611 Packet Len 64-128: 151 Packet Len 128-256: 59 Packet Len 256-1024: 199 Packet Len 1024-1500: 13562 Packet Len > 1500: 417 nDPI throughput: 27.11 K pps / 220.36 Mb/sec Analysis begin: 04/Mar/2022 14:22:52 Analysis end: 04/Mar/2022 14:23:22 Traffic throughput: 657.37 pps / 5.34 Mb/sec Traffic duration: 30.423 sec Guessed flow protos: 10 DPI Packets (TCP): 249 (8.03 pkts/flow) DPI Packets (UDP): 16 (1.07 pkts/flow) DPI Packets (other): 1 (1.00 pkts/flow) Confidence: Match by port 6 (flows) Confidence: Match by IP 4 (flows) Confidence: DPI 37 (flows) Detected protocols: DNS packets: 2 bytes: 190 flows: 1 HTTP packets: 20 bytes: 1320 flows: 3 OCSP packets: 102 bytes: 30637 flows: 6 ICMP packets: 2 bytes: 196 flows: 1 SSH packets: 70 bytes: 9884 flows: 2 YouTube packets: 19098 bytes: 20454926 flows: 14 Google packets: 585 bytes: 298742 flows: 13 TeamViewer packets: 5 bytes: 348 flows: 1 Cloudflare packets: 10 bytes: 570 flows: 1 GoogleServices packets: 87 bytes: 30436 flows: 2 AmazonAWS packets: 6 bytes: 396 flows: 1 GoogleCloud packets: 12 bytes: 792 flows: 2 Protocol statistics: Safe 30637 bytes Acceptable 342874 bytes Fun 20454926 bytes ```

Thank you guys for fixing it so fast.