search

berkeli / immersive-go

Creative Commons Zero v1.0 Universal
10 stars 0 forks source link

Batch Processing Project #23

Open berkeli opened 2 years ago

berkeli commented 2 years ago

https://github.com/CodeYourFuture/immersive-go-course/tree/main/batch-processing

  1. [x] Read the requirements
  2. [x] Implementation

Extensions:

  1. [x] The tool should write an output CSV with the URLs that failed, in a format that could be used as input: go run . --input input.csv --output output.csv --output-failed failed.csv
  2. [x] Failures in downloading and uploading the images can be temporary: introduce a retry-with-backoff strategy to make the tool more resilient to temporary failures
  3. [x] Do not re-upload the same image to S3
    • [x] Can you do this without storing anything in a database?
  4. [x] Do not download & process the same image
    • [x] Can you also do this without storing anything in a database?
  5. [x] To speed up the tool, process and upload in parallel using goroutines