Per-chunk filenames

[drzraf]

Sample (use the query hook):

query: (fileObj, chunk) => {
   // Upload "foobar.bin" first chunk at "segments/000-foobar.bin"
   chunk.filename = 'segments/' + chunk.offset.toString().padStart(3, 0) + '-' + fileObj.file.name;
}

Use-case: Flow.js can't create post DLO on Openstack Swift temporary URL but it can upload chunks.

In order to later build a DLO, each chunk must be correctly named (for sorting purposes) and all must be uploaded under a specific prefix which is possible using the File name form-data property.

This commit allows flow.js to set user-determined, per-chunk, filenames.

[AidasK]

thanks!

[command-tab]

FWIW, I've managed to PUT pieces of a Swift multi-part upload using Flow's target option:

target: (flowFile, flowChunk, isTest) => {
  const chunkNumber = flowChunk.getParams().flowChunkNumber.toString().padStart(8, '0')
  const filename = sanitizeUploadFilename(flowFile.name) // a fn that only returns alphanumerics and underscores
  return `https://swift.example.com/v1/AUTH_username/containername/${flowFile.uniqueIdentifier}/${filename}/${chunkNumber}`
}

[drzraf]

Interesting, @command-tab. In the case of temporary-url, we can't rely on PUT + auth-token because the URL (= target) itself is signed as-is for a given prefix for which POST are allowed. AFAICT the only way to provide a segment-like organization is then to play with the filenames of the form-data fields themselves instead of the target.