Requirements:
Input data format: Parquet
Output: Parquet
Error handling: implement mechanisms to handle errors
Logging and monitoring: capture relevant information for auditing and troubleshooting purposes
Lambda trigger configuration: Specify the settings required to trigger the detection of a new TXT file
Technical:
Compression: choose an appropriate compression algorithm to optimise storage
Infrastructure setup:
AWS S3 to store converted parquet files
AWS Lambda
AWS IAM roles
AWS Cloudwatch for logging and monitoring
Testing:
Unit testing: pytest
Integration testing: mock testing
Requirements: Input data format: Parquet Output: Parquet Error handling: implement mechanisms to handle errors Logging and monitoring: capture relevant information for auditing and troubleshooting purposes Lambda trigger configuration: Specify the settings required to trigger the detection of a new TXT file
Technical: Compression: choose an appropriate compression algorithm to optimise storage
Infrastructure setup: AWS S3 to store converted parquet files AWS Lambda AWS IAM roles AWS Cloudwatch for logging and monitoring
Testing: Unit testing: pytest Integration testing: mock testing