Adding a new Protocol to NTLFlowLyzer

[warlock1302]

We have collected packet capture files (pcaps) containing signaling messages between our simulated 5G core and mobile user equipment (UE). These pcaps include NGAP protocols among others.

However, when we process these pcaps using NTLFlowLyzer, no flows are generated. It appears that the NGAP protocol might not be supported by the tool.

We would like to add the new protocol to the code. Could you please provide guidance on how to add a new protocol to NTLFlowLyzer? If possible, any example code or documentation that outlines the steps required to integrate a new protocol would be extremely helpful.

Thanks in advance.

[moein-shafi]

Hi @warlock1302 Hope you are doing well. Thank you for your interest in NTLFlowLyzer and for bringing this to our attention!

I will be providing detailed guidance on how to add new protocols or even features to NTLFlowLyzer within the next couple of days.

Thank you for your patience, and please feel free to reach out if you have any further questions in the meantime.

[warlock1302]

Hi @moein-shafi Hope you are doing well. I wanted to follow up regarding the guide on adding new protocols or features to NTLFlowLyzer. Could you please provide an update on the status of this guidance?

[moein-shafi]

Hi @warlock1302,

I hope you are doing well. My apologies for the delayed response.

I have reviewed the NGAP protocol support in both the dpkt and Scapy libraries. Unfortunately, it appears that neither library currently supports parsing NGAP packets. Here are the steps to proceed based on different scenarios:

Scenario 1: Library Supports NGAP Parsing

1. Library Selection: First, choose the Python library you wish to use. We use dpkt in NTLFlowLyzer due to its superior performance compared to other libraries.

2. Implementation Steps:

   • Replace dpkt with the Selected Library:

     • Update the parts of NTLFlowLyzer where dpkt is used. This includes the network_flow_capturer/network_flow_capturer.py and network_flow_capturer/packet.py files. The packet.py file acts as a wrapper around the library packet (dpkt in this case), making it easier to switch libraries in the future and ensuring that only necessary packet information is saved to optimize memory usage.

     • Modify the network_flow_capturer.py file, which is the main parser. It reads the pcap file, iterates over all packets, and creates flows. You need to update this file to use the new library, especially focusing on the pcap_parser function. Since this function currently includes VXLAN decapsulation, I recommend rewriting it without considering the decapsulation part to simplify the transition.

     • Note that since you are using NGAP, to facilitate reusability of current features, retain the existing variable and function names in the packet and flow classes, such as get_payloadbytes, get_header_size, etc. This will help maintain consistency and reduce the effort required to update dependent features.

   • Define NGAP Flow:

     • Since NGAP is a non-TTCP-based protocol, the existing TCP flow logic in NTLFlowLyzer will not be applicable. You will need to redefine the flow logic to accommodate NGAP. Update the network_flow_capturer/flow.py and network_flow_capturer/network_flow_capturer.py files to reflect your desired flow definition for NGAP. You can keep the add_packet function and necessary getter functions, removing TCP-specific variables such as those related to bulk and sub-flows.

   • Define New Features:

     • You will need to update/change the defined features logic to accommodate the new flow definition. However, some common features related to size and timing, such as flow duration, average packet/payload/header size, inter-arrival time, etc., may remain unchanged. Ensure that the new features accurately reflect the characteristics of NGAP flows.

Scenario 2: Library Does Not Support NGAP Parsing

1. Implement NGAP Parsing: If no existing library supports NGAP, you will need to implement the parsing logic yourself. The Scapy library provides a clear method to do this, which you can find here.

2. Alternative Approach: If a library in another language (e.g., C++) supports NGAP parsing, you could write a program in that language to parse the packets and save the information in a file (e.g., JSON) or a database. You can then use Python to read this file, create flows, and extract features.

Additional Note:

Most libraries support SCTP, which might be relevant to your work:

• dpkt: dpkt SCTP support
• Scapy: Scapy SCTP support

I hope this information is helpful. Please feel free to reach out if you need any further assistance or clarification.

[moein-shafi]

Given this clarification, I'll proceed to close this particular issue. However, please don't hesitate to reconnect if you encounter any further difficulties or have additional inquiries. Your feedback is invaluable to us as we strive to maintain the integrity and functionality of NTLFlowLyzer.