search

freeCodeCamp / devdocs

API Documentation Browser
https://devdocs.io
Mozilla Public License 2.0
35.27k stars 2.36k forks source link

Uncompress s3://devdocs-downloads/foo.tar.gz to s3://devdocs-documents/foo on s3? #1639

Open simon04 opened 3 years ago

simon04 commented 3 years ago

Hi Oliver,

Uploading to s3://devdocs-documents/ takes a very long time when a lot of files are involved (for instance, dom amounts to 6852 files).

Is it feasible/possible upload only s3://devdocs-downloads/foo.tar.gz and have s3 automatically uncompress the documentation package to s3://devdocs-documents/foo? I found https://multiaki.medium.com/unzipping-s3-files-back-to-s3-without-uncompressing-entire-file-streaming-82f662b5065a, but don't know how hard/reliable this s3 setup is...

Thanks!

ojeytonwilliams commented 3 years ago

Hey Simon :wave:

That's an interesting question. I wonder if we could achieve the same effect by uploading an artifact to GitHub and triggering a workflow that unzips and syncs the result with S3. Either way, I'll take a look.

Please feel free to poke me if I seem to have forgotten, it happens!