Large File Upload Issue (Manifest File of ZERO bytes)

[azeem-sajid-confiz]

Hi,

I used the code in this section (Upload Large Files) for uploading a file of size 6 GB. The segments (i.e. 2 segments) uploaded successfully along with the manifest file (using put_object_manifest as described in the above link).

The sizes of segments is fine. But, the manifest file is of ZERO bytes. So, after this, I had three files on i.e. two segment (suffixed) files and a manifest file.

The expected behavior should be a single file (without segment files) after the successful upload.

Questions: How can I upload a large file successfully? What changes I'd be needing to achieve this in that piece of code?

Thanks!

[timuralp]

@azeem-sajid-confiz Swift doesn't have a multipart upload feature like S3 does (I think that's what you were expecting). There are two modes of uploading large files: Dynamic Large Objects and Static Large Objects (https://docs.openstack.org/swift/latest/overview_large_objects.html). From your description -- and in the sample code -- the objects are uploaded as a dynamic large object (a 0-sized manifest with a special header and prefixed segments). The SLO (static large object) way of uploading a large object is preferred (as it avoids the issues of object listings being eventually consistent, for one). In both cases, though, there will be the segments (either in the same container or a segment container) and a manifest object, which is either 0-sized (DLO) or contains a JSON description of the large object.

Hope this helps!

[azeem-sajid-confiz]

@timuralp Thank you for your response! It did help to clarify this whole concept and I was able to upload/download successfully.

However, for deletion, I was not able to find a proper API call to delete the blob. Intuitively, it should be a simple call to delete the manifest file and it should figure out the respective segments on its own.

Could you please share some pointers on this? Thanks a lot!

[timuralp]

@azeem-sajid-confiz If you're using static large objects, you can use the delete_static_large_object request to remove the SLO and its segments (https://github.com/fog/fog-openstack/blob/master/lib/fog/storage/openstack/requests/delete_static_large_object.rb). There is no support in Swift for Dynamic Large Objects. You could use the bulk delete mechanism to do so yourself, maybe? fog could also implement support for removing the associated segments, but it doesn't today.