Write PySpark Queries to Prepare Data for Planned Visualizations

[D-Mielewczyk]

Develop PySpark queries that will extract and prepare the necessary data for the visualizations detailed in the documentation. These queries should align with the planned plot types and axis configurations, ensuring that the data is correctly formatted and ready for visualization.

Requirements:

1. Review Visualization Plan:

   • Refer to the documentation that outlines the planned visualizations, plot types, and axis configurations.

2. Develop PySpark Queries:

   • Write PySpark queries to extract and transform the data for each specific plot type.
   • Ensure that the queries handle any necessary data cleaning, aggregation, and formatting.

3. Test Queries:

   • Test the PySpark queries to ensure they correctly prepare the data.
   • Validate the output data to ensure it aligns with the expected structure and content for each visualization.

4. Document Queries:

   • Document each query, explaining its purpose and how it prepares the data for the respective plot.
   • Include comments within the code to clarify each step of the query.

Details:

• Ensure that the queries are optimized for performance and handle large datasets efficiently.
• Provide clear and concise documentation and comments within the queries.
• Include instructions on how to run the queries within the EMR environment or locally.

Acceptance Criteria:

• PySpark queries that successfully prepare the data for each planned visualization.
• The output data matches the expected structure and is ready for visualization.
• Clear documentation and comments included within the queries.
• The code is committed and pushed to the GitHub repository.

Additional Notes:

• Collaborate with team members to ensure the queries meet the requirements of the planned visualizations.
• Ensure the queries are kept up-to-date with any changes to the data or visualization requirements.

[D-Mielewczyk]

Blocked by #2

[D-Mielewczyk]

and #5