Sprint 2 - Projects - Batch Processing

[bazmurphy]

Batch Processing

https://systems.codeyourfuture.io/projects/batch-processing/

Sections

1. Introduction

• Introduces the project of building an image processing pipeline that reads image URLs from a CSV file, downloads and processes the images, and uploads them to cloud storage.

2. Learning Objectives

• Lists the key learning goals of the project, including defining batch processing, using Go to run existing software, designing a CLI application, and using cloud storage technology.

3. Project Setup

• Explains how to set up and run the provided scaffolding code using Docker and Make.

4. Specification

• Outlines the requirements for the CLI tool, including reading input CSV, downloading and processing images, uploading to AWS S3, and writing output CSV.

5. How-to

• Provides guidance on implementing various aspects of the project, such as reading CSV files, downloading files, image processing, and interacting with AWS S3.

6. Extensions

• Suggests additional features and enhancements to the project, like handling failures, avoiding duplicate downloads and uploads, and parallel processing using goroutines.

Key Things To Learn:

1. Batch processing and how it differs from building servers
2. Configuring a Makefile to run a Docker image
3. Using Go to run existing software (ImageMagick) to complete tasks
4. Designing and building a CLI application to batch process images
5. Reading from and uploading data to cloud storage technology (AWS S3)
6. Reading, modifying, and extending existing code
7. Using the encoding/csv package to read and write CSV files
8. Downloading files using the http package and handling potential errors and non-200 status codes
9. Creating and configuring an AWS S3 bucket for storing processed images
10. Setting up IAM roles and policies to allow access to the S3 bucket
11. Using the AWS SDK for Go to interact with Amazon S3
12. Configuring AWS credentials and passing them to the Docker container
13. Implementing retry-with-backoff strategy for handling temporary failures in downloading and uploading images
14. Avoiding re-uploading and re-processing the same images without using a database
15. Using goroutines to process and upload images in parallel for improved performance

[SallyMcGrath]

https://systems.codeyourfuture.io/projects/batch-processing/

Give me admin access to this repo so I can edit your tickets please

[bazmurphy]

@SallyMcGrath done^ (i've edited all the links in now)