Clean Data Using PySpark for Further Analysis

[D-Mielewczyk]

Objective: Develop a PySpark script to clean the climate data fetched from the European Climate Assessment & Dataset (ECA&D). The cleaned data should be prepared and formatted appropriately for further analysis. This includes handling missing values, correcting data types, and any other necessary preprocessing steps.

Requirements:

1. The script should:
   • Load the raw data into a Spark DataFrame.
   • Handle missing values by filling, dropping, or imputing them as appropriate.
   • Ensure all data types are correct and consistent.
   • Ensure that every column is cast to the most suitable type available.
   • Normalize or transform data if needed (e.g., scaling temperature values, parsing dates).
   • Save the cleaned data to a specified location for further analysis.

Details:

• Ensure the script is well-documented and includes error handling.
• Provide comments within the script explaining each preprocessing step.
• Include instructions on how to run the script within the EMR environment or locally.

Acceptance Criteria:

• A PySpark script that successfully cleans the raw climate data and prepares it for further analysis.
• Clear documentation and instructions included in the README.md file.
• The code is committed and pushed to the GitHub repository.

Additional Notes:

• Please make sure to test the script with sample data to ensure it works as expected.
• Include any additional dependencies or setup steps required in the documentation.

Cleaning the data also involves making sure that every column is cast to the best available and most suitable type.

[D-Mielewczyk]

Not blocked at all, you can start working on this issue.