IvanNardi
commented
10 months ago Hi @vel21ripn, thanks for these interesting inputs! We are having some internal discussions about how to improve these lists, and so any feedback is welcomed!
Let' start with the real bug: the overlapping addresses... There are a few different cases:
1) Goto/citrix: we are importing the same list twice! Nice catch. I am going to remove one of them
2) MICROSOFT_AZURE vs MICROSOFT_365: these addresses are present in both the original lists (azure and ms365) explicitly provided by Microsoft itself... not sure what we should do here....
3) We don't have a MODBUS list... There are two logical separated lists in inc/generation: one list with the addresses used for protocol classification (usually used to match server address; FB, Telegram, Whatsapp,...) and one list used for flow risk detection (used to match client address; iCloudPrivateRelay, ProtonVPN exit nodes and crawlers). It is definite possible to have some addresses in both logical list
vel21ripn
Describe the bug
Some networks are described in more than one protocol.
Another problem is the lack of subnet aggregation.
To solve these problems, we need to abandon "include inc_generation/*.c.inc" and switch to automated construction of subnet lists. It also makes sense to abandon separate loading of address lists. We need to make a mask of loaded lists and one list of addresses. If I’m not mistaken, I proposed storing lists of addresses in .yaml files and collecting an optimized list of addresses from them, but for some reason the implementation was not included in nDPI. The format of the file with the list of addresses is not significant. Using '.c' files to store lists of addresses is also a good option.
The only difficulty in solving these problems is the lack of protocol names. I use a non-cross-platform solution in the form of a perl script that generates the necessary data from the ndpi_protocol_ids.h file. I don't know how much this is acceptable in an nDPI project.
I can offer my PR
Ambiguous address:
IPv4
IPv6