ncw / swift

Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)
MIT License
313 stars 107 forks source link

LargeObjectFile should support cleanup in case of upload error #99

Open majewsky opened 6 years ago

majewsky commented 6 years ago

The internal struct implementing the LargeObjectFile interface keeps track of which segments it has uploaded since {Dynamic,Static}LargeObjectCreate. When the upload fails (either during PUT of another segment or PUT of the manifest), it is useful to know which segments were already uploaded in order to perform cleanup.

Since LargeObjectFile does not have a method to expose its segments, one can only perform this cleanup by listing the uploaded segments, i.e.

names, err := conn.ObjectNameAll(segmentContainer,
  &swift.ObjectsOpts{Prefix: segmentPrefix},
)
... handle err ...
result, err := conn.BulkDelete(segmentContainer, names)

However, this is prone to eventual consistency issues: The segments might not yet be visible in the container listing. I see two possible options:

  1. Add a method like SegmentNames() []string to the LargeObjectFile interface.
  2. Add a method Cleanup() error or Reset() error to the LargeObjectFile interface that deletes all segments which were uploaded since {Dynamic,Static}LargeObjectCreate.
ncw commented 6 years ago

I like your Cleanup() error proposal. Fancy sending a PR?