Is There a Way To Export the Default nDPI Protocol Library?

[redapplesonly]

I have written my first program using nDPI, which captures live packets and sends them to nDPI for application/protocol identification. (I am working on Ubuntu 18.04, using GCC as a complier. nDPI version is 3.2)

I have the oddest issue with my code... When nDPI inspects my traffic, the ndpi_detection_process_packet() function always considers all packets to be "Unknown," meaning the struct ndpi_protocol returned by the function always looks like this:

master_protocol = 0
app_protocol = 0
category = 0

At first, I assumed my code had a bug in feeding packets to ndpi_detection_process_packet(). But then, just for fun, I added this very simple protocols file to my struct ndpi_detection_module_struct:

tcp:80@HTTP_protocol
tcp:5201@iPerf3_protocol

When I ran the code again, nDPI could successfully recognize every packet that was HTTP or iPerf3, but nothing else. Its like the protocol library within my implementation is blank, except for the protocols that I supplement with my protocols file.

I still think I must have some flaw in my implementation, and I will continue to search for it. But I can't help but ask... if my code will only recognize protocols listed in the protocol file, can I make use of that as a workaround? Is there some way to export nDPI's default protocol file, which I could then feed to my code?

Unfortunately, I cannot post my project to this forum. I suspect that my code's problem is in my struct ndpi_detection_module_struct, and I'll post the code that initializes it below. It should look familiar, as it was, ah, "borrowed" from ndpiReader.c. Thank you for any advice you can offer.

static struct ndpi_detection_module_struct *ndpi_info_mod = NULL;

int main(int argc, char ** argv)
{

    ndpi_init_prefs init_prefs = ndpi_no_prefs;
    ndpi_info_mod = ndpi_init_detection_module( ndpi_no_prefs );
    if(ndpi_info_mod == NULL) return -1;

    NDPI_PROTOCOL_BITMASK all_protocols;
    NDPI_BITMASK_SET_ALL( all_protocols );
    ndpi_set_protocol_detection_bitmask2( ndpi_info_mod, &all_protocols );
    ndpi_load_protocols_file( ndpi_info_mod, "/home/me/simple.protos" );
    ndpi_finalize_initalization( ndpi_info_mod );       // Init the Detect Mod

    ...more code here...
}

[lucaderi]

Besides the fact that you don't check any return value, the code looks good so far

[redapplesonly]

Thanks Luca, I'll revise the code to check the return values; perhaps that will show why my detection module can't recognize any non-customized protocols. I hope so. :)

Is there no way to export the non-customized (default) protocols into a protocol file? Many thanks!

[aouinizied]

@redapplesonly this is not possible and will not solve your issue. Your issue root cause is probably how you feed packet to process function or how you build your packet to flow aggregation.

• The fact that detection is working with your proto file is due to simple rules you added: TCP port 80 is HTTP and TCP port 5201 is Iperf. Your rules are valid even without flow aggregation as can be detected using single packet headers only.
• nDPI core dissectors (most of them) are based pure payload packet dissection logic and cannot be dumped in such a set of matching rules. The proto file interface is intended to extend nDPI with additional hostname/port/proto user defined matching rules.

If you cannot share your code example here, I advice you to have a look at ndpiSimpleIntegration.c under example directory and make sure you follow the same logic. The code is simpler than reader example as a starting point.

Here some hints:

• Check that your packet parsing and flow construction is valid.
• Check that for each flow you maintain an ndpi_flow_struct and ndpi_src_id and ndpi_dst_id references.
• Make sure that when you feed process_packet you switch src and dst id struct based on flow direction.
• Check that you do not interrupt your dissection logic too early.

Hope this helps.

BR

[redapplesonly]

Excellent, thank you, BR. This is solid, useful advice, and I appreciate you taking the time to explain it to me, far more than you can know. I'll work through your checklist. Thank you!