File got locked even after uploaded successfully in retries after getting error

[suhas2390]

Describe the bug I have a node service which uploads files to server. As per the documentation, I am using the filestream in tus client and a chunkSize defined is 300MB. Also, it retries 5 times if any error occurs from server / network. If I receive any error during upload, it retries and upload file successfully. But file gets locked by service. As per the documentation, I haven't destroy the stream on success. If there isn't any error-retry occured while uploading file, it got released. I am deleting a file from folder once it is uploaded. This is how I got to know that it is locked.

To Reproduce You can run below code with lower internet upstream speed (~200kbps , so that you can get some error or network disconnection) and start uploading 3GB file. It will take around 6-7 hours to upload and hope you get an error.

Please try to reproduce the problem when uploading to our public tus instance at https://tusd.tusdemo.net/files/, if applicable to your situation. This helps us to determine whether the problem may be caused by the server component.

const fileDataStream = fs.createReadStream(filepath);
  const stats = fs.statSync(filepath);
  const fileSizeInBytes = stats['size']; 
  let tusClient = new tus.Upload(fileDataStream, {
        endpoint: 'my-server-url',
        headers: {
          Authorization: 'Bearer ' + apiKey,
          'Upload-Length': fileSizeInBytes,
        },
        chunkSize: 1024 * 1024 * 300, // 300MB
        retryDelays: [0, 3000, 5000, 10000, 20000],
        metadata: { filename: fileName },
        onError: function (error) {
          console.log(error);
          fileDataStream.destroy();
          removeFileSync(filepath);
        },
        onShouldRetry: function (err, retryAttempt) {
          console.log(`err: ${err} retryattempt: ${retryAttempt}`)
          return true;
        },
        onSuccess: async function () {
          fs.unlinkSync(filepath);
        }
      });

      tus.start();

Expected behavior A file should be released once a upload is completed successfully in any case.

[Acconut]

I am deleting a file from folder once it is uploaded. This is how I got to know that it is locked.

What exactly do you mean by saying "locked" here? Is the upload resource on the server locked (i.e. tus server responds with 423 status) or is tus-js-client still holding an open file descriptor to the file? What is the exact error you are getting?

[suhas2390]

Upload resource on client got locked. File could not be deleted if I try to delete it. So could not do any operation on file after I got an error. My server responds with different errors other than 423. ex. 400, 500, socket hangup, network failure..

[Acconut]

Ok, I understand. What OS are you using?

I am wondering if the behavior of the autoClose option from fs.createReadStream is different than we though. We do not close the streams manually but let autoClose handle it. Maybe that is not sufficient for error cases.