B2.4 Split dataset into training and validation sets

[BillyHeYifan]

• [x] Split the integrated dataset into a training set and a validation set.
• [x] Use a 9:1 ratio, where 90% of the data is allocated to the training set and 10% to the validation set.
• [x] Ensure the data is shuffled before splitting to avoid any potential bias in the dataset.
• [x] Perform checks to verify the correct distribution and integrity of the split data.
• [x] Document the process and ensure consistency across different data splits.

[ZhihengShen]

The process of splitting the integrated dataset into training and validation sets was executed effectively, adhering to the specified 9:1 ratio. The shuffling step ensured that the data was randomized, reducing potential bias and allowing for a more accurate evaluation of the model. The distribution between the training and validation sets was verified, and integrity checks confirmed that the data was split consistently without any loss or duplication of records. The documentation provided a clear outline of the steps taken, ensuring reproducibility across different splits. Overall, this task was completed with attention to detail and a strong focus on maintaining data integrity throughout the process.