Data Standardisation and Cleaning

DRedDevil04 commented 10 months ago

Data Standardization and Cleaning Notebook

Issue Description

📄 Task: Develop a Python notebook to facilitate the integration of three provided datasets. The goal is to standardize labels and features, ensuring consistency across all datasets. Additionally, implement data cleaning procedures to enhance data quality. Finally, store the processed data in variables X (features) and Y (labels), with labels being binary (0 or 1) to represent the presence or absence of a DDoS attack.

Implementation Steps

Data Reading:
- Create functions to read data from the three datasets, considering their respective formats.
Standardization:
- Standardize labels and features across datasets to ensure uniformity.
- Align features to match their corresponding counterparts in other datasets.
- Then CONCATENATE all three datasets
Data Cleaning:
- Implement cleaning procedures to handle missing values, outliers, or any inconsistencies in the datasets.
Label Standardization:
- Convert labels to binary values (0 or 1) based on the presence or absence of DDoS attacks.
Variable Assignment:
- Store the standardized and cleaned data in variables, assigning features to variable X and labels to variable Y.

Additional Considerations

Documentation:
- Provide inline comments and documentation for each step to enhance code readability.
Testing:
- Include testing steps to ensure the notebook produces the expected results.
Guidelines:
- Adhere to the project's coding and documentation guidelines.