How to clean up partial upload chunks after final upload success?

[loren-m-crawford]

Question Hey folks! I'm having a bit of a difficult time understanding how best to clean up partial upload chunks from our datastore after we've successfully uploaded the final concatenated upload. Currently, our partial uploads and final uploads are getting stored to the same bucket and aren't tagged differently or getting cleaned up. Do we need to add something different in our config? Do we need to have a different ConcaterDataStore for partials to get cleaned up properly?

Setup details Please provide following details, if applicable to your situation:

• Operating System: Linux
• Used tusd version: v2.4.0
• Used tusd data storage: AWS S3
• Used tusd configuration:

  composer := tusd.NewStoreComposer()
  composer.UsesConcater = true
  config := tusd.Config{
      MaxSize:                 26 * 1024 * 1024 * 1024,
      BasePath:                "/files/",
      RespectForwardedHeaders: true,
      StoreComposer:           composer, 
  })

• Used tus client library: tus-js-client

[loren-m-crawford]

I think I understand what I need to do. I believe it looks like I need to setup another S3Store for the ConcaterStore on the composer with a separate bucket path with a different delete policy.

[Acconut]

With tus, a partial upload could be reused for a different concatenation in theory, so they are not cleaned up automatically. When we used S3 as our backend in production, we had a lifetime policy which removed all old uploads, which included partial and final uploads. Depending on your use case, it might be fine to use a similar approach and handle the lifetime of partial uploads just like any other upload.

Alternatively, you could use a post-finish hook to manually delete the partial uploads that were used for concatenation. The hook request includes information about whether the upload is a final or partial part and what partial uploads where used (see IsFinal, IsPartial, and PartialUploads).

I hope this helps.