search

noobaa / noobaa-core

High-performance S3 application gateway to any backend - file / s3-compatible / multi-clouds / caching / replication ...
https://www.noobaa.io
Apache License 2.0
271 stars 80 forks source link

New feature - Auto expose contents inside Archive Objects (Tar,Zip,7z) through normal S3 API #8148

Open guymguym opened 5 months ago

guymguym commented 5 months ago

Is your feature request related to a problem?

Uploading a lot of small objects is slow and wasteful. Instead, uploading a large tar or zip archives is better. If only the S3 endpoint could make it look like the objects are individual even though it was uploaded as a single archive, and without having to expand the file, only access it on demand.

Describe the solution you'd like

Head/Get Object will access directly to files inside the archive object with keys like /path/to/archive.zip/path/inside/file.txt ListObjects should be able to list files inside the archive object with prefix of /path/to/archive.zip/

Describe alternatives you've considered

Another option is expand it during the upload, but I would prefer to keep it as a single object and just allow to access it, because this also optimizes the storage density of such small objects.

Additional context

It would be great fi this can be implemented in the S3 endpoint layer and doesn't need to be implemented differently for every namespace, although there are definitely challenges to do that...

github-actions[bot] commented 2 months ago

This issue had no activity for too long - it will now be labeled stale. Update it to prevent it from getting closed.