Detect 5 Areas with the Biggest Temperature Changes Using PySpark

[D-Mielewczyk]

Develop a PySpark script to analyze the cleaned climate data and identify the 5 areas with the most significant temperature changes. The script should be saved with an appropriate name reflecting its functionality.

Requirements:

1. Load the cleaned data:

   • Use PySpark to load the preprocessed and cleaned climate data from the specified location.

2. Data Analysis:

   • Analyze the temperature data to detect the 5 areas with the largest temperature changes over the specified time period.
   • Ensure the analysis is accurate and considers all relevant factors to identify the areas with the most significant changes.

3. Output:

   • Save the results of the analysis, including the names of the 5 areas and their respective temperature changes, to a specified location.

4. Script Naming:

   • Save the PySpark script with a fitting name such as detect_top5_temperature_changes.py.

Details:

• Ensure the script is well-documented and includes error handling.
• Provide comments within the script explaining each step of the analysis.
• Include instructions on how to run the script within the EMR environment or locally.

Acceptance Criteria:

• A PySpark script (detect_top5_temperature_changes.py) that successfully identifies the 5 areas with the most significant temperature changes.
• The script outputs the results to a specified location.
• Clear documentation and instructions included in the README.md file.
• The code is committed and pushed to the GitHub repository.

Additional Notes:

• Please make sure to test the script with sample data to ensure it works as expected.
• Include any additional dependencies or setup steps required in the documentation.

[D-Mielewczyk]

Blocked by #2