minio / minio-js

MinIO Client SDK for Javascript
https://docs.min.io/docs/javascript-client-quickstart-guide.html
Apache License 2.0
963 stars 282 forks source link

I find a bug, putObject can not resume upload when use multipart upload , see the source code #1318

Closed juoliii closed 4 months ago

juoliii commented 4 months ago

private async uploadStream( bucketName: string, objectName: string, headers: RequestHeaders, body: stream.Readable, partSize: number, ): Promise { // A map of the previously uploaded chunks, for resuming a file upload. This // will be null if we aren't resuming an upload. const oldParts: Record<number, Part> = {}

// Keep track of the etags for aggregating the chunks together later. Each
// etag represents a single chunk of the file.
const eTags: Part[] = []

const previousUploadId = await this.findUploadId(bucketName, objectName)
let uploadId: string
if (!previousUploadId) {
  uploadId = await this.initiateNewMultipartUpload(bucketName, objectName, headers)
} else {
  uploadId = previousUploadId
  const oldTags = await this.listParts(bucketName, objectName, previousUploadId)
  oldTags.forEach((e) => {
    oldTags[e.part] = e
  })
}

in the source code , oldParts aways empty

prakashsvmx commented 4 months ago

can you share mc admin trace -v <ALIAS> for the complete multipart upload/resume ?

juoliii commented 4 months ago

can you share mc admin trace -v <ALIAS> for the complete multipart upload/resume ?

please see the source code client.ts , line 1698,the var oldParts, aways empty,this is a bug , and putObject can not resume upload

harshavardhana commented 4 months ago

There is no resume functionality in multipart API.

Relying on listparts() is racy as it's point in time consistent it is not aware of concurrent PUTs on the same uploadId.

And there is no serialization between both. So relying on that to finish complete multipart can potentially lead to corruption.

You should manage what you uploaded locally if you wish to resume disparate parts.

There is no proper way to resume in S3 API which is safe on the server side. It is designed for atomic uploads.