kumadownloader

[peterbe]

Here's how this is different from: aws s3 sync s3://mdn-api-prod/ mdn-api-prod/

• It uses exact ETags so if a file changes by 0 bytes (characters remove == characters added) it will notice that the file has changed.
• Kuma's Document JSON API S3 files have key names that work in S3 but won't work on disk such as file names or directory names that are too long.
• It yields a file called _new_files.txt which forms the foundation for re-running exclusively the new/changed files into the pipeline.