Application Layer Flow Analyzer (ALFlowLyzer)

As part of the Understanding Cybersecurity Series (UCS), ALFlowLyzer is a Python open-source project to extract application layer features from network traffic for Anomaly Profiling (AP) which is the third component of the NetFlowLyzer.

ALFlowLyzer generates bidirectional flows from the Application Layer of network traffic, where the first packet determines the forward (source to destination) and backward (destination to source) directions, hence the statistical time-related features can be calculated separately in the forward and backward directions. Additional functionalities include selecting features from the list of existing features, adding new features, and controlling the duration of flow timeout. In the first version, it supports DNS protocol and in the next versions, other protocols will be supported. For more information regarding the DNS flow definition, please refer to the corresponding paper in the Copyright section.

Table of Contents

• Installation
• Execution
  • Configuration File
  • Argument Parser
• Architecture
• Extracted Features
  • DNS Related
  • Statistical Information Calculation
• Output
• Copyright (c) 2024
• Contributing
• Project Team members
• Acknowledgement

Installation

Before installing or running the ALFlowLyzer package, it's essential to set up the necessary requirements on your system. Begin by ensuring you have both Python and pip installed and functioning properly (execute the pip3 --version command). Then, execute the following command:

pip3 install -r requirements.txt

You are prepared to install ALFlowLyzer. To proceed, execute the following command in the package's root directory (where the setup.py file is located), which will install the ALFlowLyzer package on your system:

On Linux:

python3 setup.py install

On Windows:

pip3 install .

After successfully installing the package, confirm the installation by running the following command:

alflowlyzer -h

usage: ALFlowLyzer [-h] [-c CONFIG_FILE] [-o] 
options:
 -h, --help            show this help message and exit
 -c CONFIG_FILE, --config-file CONFIG_FILE
                       JSON config file address.
 -o, --online-capturing
                       Capturing mode. The default mode is offline capturing. 

Execution

The core aspect of running ALFlowLyzer involves preparing the configuration file. This file is designed to facilitate users in customizing the program's behavior with minimal complexity and cost, thus enhancing program scalability. Below, we outline how to prepare the configuration file and subsequently demonstrate how to execute ALFlowLyzer using it.

Configuration File

The configuration file is formatted in JSON, comprising key-value pairs that enable customization of the package. While some keys are mandatory, others are optional. Below, each key is explained along with its corresponding value:

• pcap_file_address [Required]

  This key specifies the input PCAP file address. The format of the value should be a string.

  Note: At this version of ALFlowLyzer, we only support the PCAP format. For other formats such as PCAPNG, you must convert them to PCAP. To convert PCAPNG to PCAP, you can use Wireshark. If you prefer command-line tools, you can use the following command:

  tshark -F pcap -r {pcapng_file} -w {pcap_file}

  Replace {pcapng_file} with the path to your PCAPNG file and {pcap_file} with the desired output PCAP file name.

• output_file_address [Required]

  This key specifies the output CSV file address. The format of the value should be a string.

• label [Optional]

  This key specifies the value of the label column in the output CSV file address. The format of the value should be a string. The default value is Unknown.

• number_of_threads [Optional]

  This key specifies the number of threads to be used for all processes, including flow extraction, feature calculation, and output writing. The value must be an integer of at least 3. The default value is 4.

  It's important to consider that the optimal value for this option varies based on the system configuration and the format of the input PCAP file. For instance, if the PCAP file contains a large number of packets (e.g., more than 5 million) and they are all TCP packets, increasing the number of threads might be beneficial. However, if the packets represent a small number of flows and all related packets are contiguous, adding more threads could potentially slow down the program since there are fewer distinct flows.

  As a rule of thumb, the ideal value for this option typically falls between half the number of CPU cores (CPU count) and twice the CPU count. This helps balance computational resources without overwhelming the system. (0.5 * cpu_count < best_option < 2 * cpu_count)

• feature_extractor_min_flows [Optional]

  This key determines the minimum number of finished flows required for the feature extractor thread to initiate its work and extract features from these finished flows. The value must be an integer. The default value is 4000.

  Selecting a high value for this option will consume more RAM since more flows will be stored in memory, potentially slowing down the entire program. Conversely, choosing a low value for this option can slow down the execution process, as it involves locking the finished flows list and then copying those flows for feature extraction. These two processes, locking and copying, are slow and can impede other program components.

• writer_min_rows [Optional]

  This key specifies the minimum number of ready flows (i.e., finished flows from which features have been extracted) required for the writer thread to begin its work of writing the flows to the CSV file. The value must be an integer. The default value is 6000.

  Opting for a high value for this option will increase RAM usage since more flows will be stored in memory, potentially slowing down the overall program performance. Conversely, selecting a low value for this option can slow down the execution process, involving locking the finished flows list, copying those flows for the writing process, and performing I/O operations to write to the file. These three processes — locking, copying, and I/O — are slow and may impede other program components.

• read_packets_count_value_log_info [Optional]

  This key determines the minimum number of processed packets (i.e., the number of packets read from the PCAP file and assigned to a flow) required for the logger to log. The value must be an integer. The default value is 10,000. This means that after processing every 10,000 packets, the program will print a statement indicating the number of packets analyzed.

• check_flows_ending_min_flows [Optional]

  This key specifies the minimum number of ongoing flows (i.e., created flows that have not yet finished) required for checking if they have reached the timeout or maximum flow time value. The value must be an integer. The default value is 2000. This indicates that if the number of ongoing flows exceeds 2000, the program will proceed to check all flows for timeout or maximum flow time.

• capturer_updating_flows_min_value [Optional]

  This key determines the minimum number of finished flows required to be added to the queue for feature extraction. The value must be an integer. The default value is 2000. This means that if the number of finished flows exceeds 2000, the program will move them to a separate list for the feature extractor.

• max_flow_duration [Optional]

  This key sets the maximum duration of a flow in seconds. The value must be an integer. The default value is 120,000. It means if the flow duration exceeds 120,000 seconds, the program will terminate the flow and initiate a new one.

• activity_timeout [Optional]

  This key defines the flow activity timeout in seconds. The value must be an integer. The default value is 5000. It means if 5000 seconds have elapsed since the last packet of the flow, the program will terminate the flow.

• floating_point_unit [Optional]

  This key specifies the floating point unit used for the feature extraction process. The value must be in the format: .[UNIT]f. The default value is .4f. This indicates that the feature values will be rounded to the fourth decimal place.

• max_rows_number [Optional]

  This key defines the maximum number of rows in the output CSV file. The value must be an integer. The default value is 900,000. It means if there are more than 900,000 flows to be written in the CSV file, the program will close the current CSV file and create a new one for the remaining flows.

• features_ignore_list [Optional]

  This key specifies the features that you do not want to extract. The value must be a list of string values, where each string represents a feature name. The default value is an empty list. If you include a feature name in this list, the program will skip extracting that feature, and it will not appear in the output CSV file.

An example of a configuration file would be like this:

{
    "pcap_file_address": "/mnt/c/dataset/my_pcap_file.pcap",
    "output_file_address": "./output-of-my_pcap_file.csv",
    "label": "Benign",
    "number_of_threads": 4,
    "feature_extractor_min_flows": 2500,
    "writer_min_rows": 1000,
    "read_packets_count_value_log_info": 1000000,
    "check_flows_ending_min_flows": 20000,
    "capturer_updating_flows_min_value": 5000,
    "dns_activity_timeout": 30,
    "max_flow_duration": 120000,
    "floating_point_unit": ".4f",
    "max_rows_number": 800000,
    "features_ignore_list": [
        "dns_whois_domain_name",
        "dns_domain_email",
        "dns_domain_registrar",
        "dns_domain_creation_date",
        "dns_domain_expiration_date",
        "dns_domain_age",
        "dns_domain_country",
        "dns_domain_dnssec",
        "dns_domain_dnssec",
        "dns_domain_address",
        "dns_domain_city",
        "dns_domain_state",
        "dns_domain_zipcode",
        "dns_domain_name_servers",
        "dns_domain_updated_date"
    ]
}

In general, we recommend adjusting the values of the following options: number_of_threads, feature_extractor_min_flows, writer_min_rows, check_flows_ending_min_flows, and capturer_updating_flows_min_value, based on your system configuration. This is particularly important if your PCAP file is large (usually more than 4 GB with over 1 million TCP packets), to optimize program efficiency.

Argument Parser

You can use -h to see different options of the program.

To execute ALFlowLyzer, simply run the following command:

alflowlyzer -c YOUR_CONFIG_FILE

Replace YOUR_CONFIG_FILE with the path to your configuration file.

Moreover, this project has been successfully tested on Ubuntu 20.04, Ubuntu 22.04, Windows 10, and Windows 11. It should work on other versions of Ubuntu OS (or even Debian OS) as long as your system has the necessary Python3 packages (you can find the required packages listed in the requirements.txt file).

Architecture

Extracted Features

We currently have currently 130 features that are as follows:

1. Duration
2. Packets Numbers
3. Receiving Packets Numbers
4. Sending Packets Numbers
5. Successful packet numbers (HTTP packets only)
6. Successful packet rate (HTTP packets only)
7. Delta Start
8. Handshake Duration
9. Total Bytes
10. Receiving Bytes
11. Sending Bytes
12. Packets Rate
13. Receiving Packets Rate
14. Sending Packets Rate
15. Packets Len Rate
16. Receiving Len Packets Rate
17. Sending Len Packets Rate
18. Packets Len Min
19. Packets Len Max
20. Packets Len Mean
21. Packets Len Median
22. Packets Len Mode
23. Packets Len Standard Deviation
24. Packets Len Variance
25. Packets Len Coefficient of Variation
26. Packets Len Skewness
27. Receiving Packets Len Min
28. Receiving Packets Len Max
29. Receiving Packets Len Mean
30. Receiving Packets Len Median
31. Receiving Packets Len Mode
32. Receiving Packets Len Standard Deviation
33. Receiving Packets Len Variance
34. Receiving Packets Len Coefficient of Variation
35. Receiving Packets Len Skewness
36. Sending Packets Len Min
37. Sending Packets Len Max
38. Sending Packets Len Mean
39. Sending Packets Len Median
40. Sending Packets Len Mode
41. Sending Packets Len Standard Deviation
42. Sending Packets Len Variance
43. Sending Packets Len Coefficient of Variation
44. Sending Packets Len Skewness
45. Receiving Packets Delta Len Min
46. Receiving Packets Delta Len Max
47. Receiving Packets Delta Len Mean
48. Receiving Packets Delta Len Median
49. Receiving Packets Delta Len Standard Deviation
50. Receiving Packets Delta Len Variance
51. Receiving Packets Delta Len Mode
52. Receiving Packets Delta Len Coefficient of Variation
53. Receiving Packets Delta Len Skewness
54. Sending Packets Delta Len Min
55. Sending Packets Delta Len Max
56. Sending Packets Delta Len Mean
57. Sending Packets Delta Len Median
58. Sending Packets Delta Len Standard Deviation
59. Sending Packets Delta Len Variance
60. Sending Packets Delta Len Mode
61. Sending Packets Delta Len Coefficient of Variation
62. Sending Packets Delta Len Skewness
63. Receiving Packets Delta Time Max
64. Receiving Packets Delta Time Mean
65. Receiving Packets Delta Time Median
66. Receiving Packets Delta Time Standard Deviation
67. Receiving Packets Delta Time Variance
68. Receiving Packets Delta Time Mode
69. Receiving Packets Delta Time Coefficient of Variation
70. Receiving Packets Delta Time Skewness
71. Sending Packets Delta Time Min
72. Sending Packets Delta Time Max
73. Sending Packets Delta Time Mean
74. Sending Packets Delta Time Median
75. Sending Packets Delta Time Standard Deviation
76. Sending Packets Delta Time Variance
77. Sending Packets Delta Time Mode
78. Sending Packets Delta Time Coefficient of Variation
79. Sending Packets Delta Time Skewness

note: Delta features are about differences (time or length or anything else) between two 'consecutive' packets.

DNS Related

1. Domain Name
2. WhoisDomainName
3. Top Level Domain
4. Second Level Domain
5. Domain Name Length
6. Sub Domain Name Length
7. Domain Name 1-Gram
8. Domain Name 2-Gram
9. Domain Name 3-Gram
10. Numerical Percentage
11. Character Distribution
12. Character Entropy
13. DomainEmail
14. DomainRegistrar
15. DomainCreationDate
16. DomainExpirationDate
17. DomainAge
18. DomainCountry
19. DomainDNSSEC
20. DomainOrganization
21. DomainAddress
22. DomainCity
23. DomainState
24. DomainZipcode
25. DomainNameServers
26. DomainUpdatedDate
27. Continuous Numeric Max Len
28. Continuous Alphabet Max Len
29. Continuous Consonant Max Len
30. Continuous Same Alphabet Max Len
31. Vowel Consonant Ratio
32. Conv Freq Vowel Consonant
33. Distinct TTL Values
34. TTL Values Min
35. TTL Values Max
36. TTL Values Mean
37. TTL Values Mode
38. TTL Values Variance
39. TTL Values Standard Deviation
40. TTL Values Median
41. TTL Values Skewness
42. TTL Values Coefficient of Variation
43. Distinct A Resource Records
44. Distinct NS Resource Records
45. Average Authority Resource Records
46. Average Additional Resource Records
47. Average Answer Resource Records
48. Query Resource Record Type
49. Answer Resource Record Type
50. Query Resource Record Class
51. Answer Resource Record Class

Statistical Information Calculation

We use differnet libraries to calculate various mathematical equations. Below you can see the libraries and their brief definition based on their documentations:

• statistics

  This module provides functions for calculating mathematical statistics of numeric (Real-valued) data.

  The module is not intended to be a competitor to third-party libraries such as NumPy, SciPy, or proprietary full-featured statistics packages aimed at professional statisticians such as Minitab, SAS and Matlab. It is aimed at the level of graphing and scientific calculators.

• scipy

  SciPy is a third-party library for scientific computing based on NumPy. It offers additional functionality compared to NumPy, including scipy.stats for statistical analysis. In this project, we use 'scipy.stats'.

Nine mathematical functions are used to extract different features. You can see how those functions are calculated in the ALFlowLyzer below:

1. Min

   You know what it means :). The 'min' function (Python built-in) calculates the minimum value in a given list.

2. Max

   Same as min. The 'max' function (Python built-in) calculates the minimum value in a given list.

3. Mean

   The 'mean' function from 'statistics' library (Python built-in) calculates the mean value of a given list. According to the library documentation:

   The arithmetic mean is the sum of the data divided by the number of data points. It is commonly called “the average”, although it is only one of many different mathematical averages. It is a measure of the central location of the data.

   This runs faster than the mean() function and it always returns a float. The data may be a sequence or iterable. If the input dataset is empty, raises a StatisticsError.

4. Median

   The 'median' function from 'statistics' library (Python built-in) calculates the mean value of a given list. According to the library documentation:

   Return the median (middle value) of numeric data, using the common “mean of middle two” method. If data is empty, StatisticsError is raised. data can be a sequence or iterable.

   The median is a robust measure of central location and is less affected by the presence of outliers. When the number of data points is odd, the middle data point is returned. When the number of data points is even, the median is interpolated by taking the average of the two middle values:

5. Variance

   The 'pvariance' function from 'statistics' library (Python built-in) calculates the mean value of a given list. According to the library documentation:

   Return the population variance of data, a non-empty sequence or iterable of real-valued numbers. Variance, or second moment about the mean, is a measure of the variability (spread or dispersion) of data. A large variance indicates that the data is spread out; a small variance indicates it is clustered closely around the mean.

   Raises StatisticsError if data is empty.

6. Standard Deviation

   The 'pstdev' function from 'statistics' library (Python built-in) calculates the mean value of a given list. According to the library documentation:

   Return the population standard deviation (the square root of the population variance). See pvariance() for arguments and other details.

7. Mode

   The 'mode' function from 'scipy.stats' library calculates the mode value of a given list. According to the library documentation, this function:

   Return an array of the modal (most common) value in the passed array.

   If there is more than one such value, only the smallest is returned. The bin-count for the modal bins is also returned.

8. Coefficient of Variation

   The 'variation' function from 'scipy.stats' library calculates the mode value of a given list. According to the library documentation, this function:

   The coefficient of variation is the standard deviation divided by the mean.

   There are several edge cases that are handled without generating a warning:

   • If both the mean and the standard deviation are zero, nan is returned.

   • If the mean is zero and the standard deviation is nonzero, inf is returned.

   • If the input has length zero (either because the array has zero length, or all the input values are nan and nan_policy is 'omit'), nan is returned.

   • If the input contains inf, nan is returned.

9. Skewness

   The 'skew' function from 'scipy.stats' library calculates the mode value of a given list. According to the library documentation, this function:

   For normally distributed data, the skewness should be about zero. For unimodal continuous distributions, a skewness value greater than zero means that there is more weight in the right tail of the distribution.

   The sample skewness is computed as the Fisher-Pearson coefficient of skewness, i.e.

   

   where

   

   is the biased sample ith central moment, and x- is the sample mean. If bias is False, the calculations are corrected for bias and the value computed is the adjusted Fisher-Pearson standardized moment coefficient, i.e.

   

Output

Copyright (c) 2024

For citation in your works and also understanding ALFlowLyzer completely, you can find below published papers:

• “Unveiling malicious DNS behavior profiling and generating benchmark dataset through application layer traffic analysis”, MohammadMoein Shafi, Arash Habibi Lashkari, and Hardhik Mohanty, Computers and Electrical Engineering, Vol 118, 2024

Contributing

Any contribution is welcome in the form of pull requests.

Project Team members

• Arash Habibi Lashkari: Founder and supervisor

• Moein Shafi: Graduate student, Researcher and developer - York University

• Hardik Mohanty: Mitacs Global Research Internship (GRI), Researcher and developer - York University

Acknowledgement

This project has been made possible through funding from the Natural Sciences and Engineering Research Council of Canada — NSERC (#RGPIN-2020-04701), Canada Research Chair (Tier II) - (#CRC-2021-00340) to Arash Habibi Lashkari and Mitacs Global Research Internship (MGRI) program for summer student.