search

veliovgroup / Meteor-Files

🚀 Upload files via DDP or HTTP to ☄️ Meteor server FS, AWS, GridFS, DropBox or Google Drive. Fast, secure and robust.
https://packosphere.com/ostrio/files
BSD 3-Clause "New" or "Revised" License
1.11k stars 166 forks source link

ENOENT: no such file or directory, mkdir #857

Closed Leekao closed 1 year ago

Leekao commented 1 year ago

I'm having an issue uploading a file, after the file upload begins I get an exception on the server with an error in the mkdirSync in write-stream.js, I'm using ostrio:files@2.3.0 and my Meteor is .2.8.0, I'm developing on windows 11 and using chrome Version 107.0.5304.87 (Official Build) (64-bit), this is clearly a Server issue, this is the log on the server:

I20221102-21:45:49.168(2)? [FilesCollection] [File Start Method] output3.wav - KEL7EyEGCN9eA4p2m
I20221102-21:45:49.168(2)? [FilesCollection] [Upload] [DDP Start Method] Got #-1/1 chunks, dst: output3.wav
I20221102-21:45:49.199(2)? Exception in callback of async function: Error: ENOENT: no such file or directory, mkdir
I20221102-21:45:49.200(2)?     at Object.mkdirSync (fs.js:1014:3)
I20221102-21:45:49.201(2)?     at packages/ostrio:files/write-stream.js:47:16
I20221102-21:45:49.205(2)?     at packages/ostrio:files/write-stream.js:10:50
I20221102-21:45:49.206(2)?     at runWithEnvironment (packages\meteor.js:1347:24) {
I20221102-21:45:49.206(2)?   errno: -4058,
I20221102-21:45:49.207(2)?   syscall: 'mkdir',
I20221102-21:45:49.207(2)?   code: 'ENOENT'
I20221102-21:45:49.208(2)? }
I20221102-21:45:49.221(2)? [FilesCollection] [Upload] [DDP] Got #1/1 chunks, dst: output3.wav
I20221102-21:45:49.226(2)? [FilesCollection] [Upload] [DDP] Got #-1/1 chunks, dst: output3.wav

My storagePath is set to a directory outside the project, if that matters.

This is the log from the client:

[FilesCollection] [insert()]
core.js:108 [FilesCollection] [FileUpload] [constructor]
core.js:108 [FilesCollection] [insert] using WebWorkers
upload.js:368 loadFile output3.wav: 49.4609375 ms
core.js:108 [FilesCollection] [UploadInstance] [sendEOF] false
Hello.jsx:13 {outputFile: File}
core.js:108 [FilesCollection] [insert()]
upload.js:197 Timer 'insert output3.wav' already exists
UploadInstance @ upload.js:197
insert @ client.js:258
onComplete @ Hello.jsx:14
(anonymous) @ Hello.jsx:30
invokePassiveEffectCreate @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:27450
callCallback @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:7908
invokeGuardedCallbackDev @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:7957
invokeGuardedCallback @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:8019
flushPassiveEffectsImpl @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:27537
unstable_runWithPriority @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:30769
runWithPriority$1 @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:15239
flushPassiveEffects @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:27410
performSyncWorkOnRoot @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:26232
(anonymous) @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:15290
unstable_runWithPriority @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:30769
runWithPriority$1 @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:15239
flushSyncCallbackQueueImpl @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:15285
flushSyncCallbackQueue @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:15272
scheduleUpdateOnFiber @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:25856
dispatchAction @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:20102
(anonymous) @ modules.js?hash=2bdaaa6ac539e0da317c883b0d42b092fade204e:32283
core.js:108 [FilesCollection] [FileUpload] [constructor]
core.js:108 [FilesCollection] [insert] using WebWorkers
upload.js:368 loadFile output3.wav: 45.8408203125 ms
core.js:108 [FilesCollection] [UploadInstance] [sendEOF] false
onurcansevinc commented 1 year ago

I have same issue too

dr-dimitru commented 1 year ago

@Leekao , error means storagePath not readable or doesn't exists

onurcansevinc commented 1 year ago

@Leekao , error means storagePath not readable or doesn't exists

I didn't change/define the storagePath, so it's default

Leekao commented 1 year ago

@Leekao , error means storagePath not readable or doesn't exists

The directory was created on C:\ by the constructor, I can see it. @dr-dimitru I see you tagged "Need more info", what more info do you need from me?

dr-dimitru commented 1 year ago

@Leekao

  1. not followed our ISSUE_TEMPLATE
  2. Server logs are not full and should output storagePath you've set at the beginning
  3. Seems like a simple issue related to usage on Windows, not related to the package
Leekao commented 1 year ago

@Leekao

  1. not followed our ISSUE_TEMPLATE
  2. Server logs are not full and should output storagePath you've set at the beginning
  3. Seems like a simple issue related to usage on Windows, not related to the package

Here is the full log I have:

I20221103-03:15:02.402(2)? [FilesCollection.storagePath] Set to: C:\AOC_tmp\wavs
I20221103-03:15:02.419(2)? ** You've set up some data subscriptions with Meteor.publish(), but
I20221103-03:15:02.419(2)? ** you still have autopublish turned on. Because autopublish is still
I20221103-03:15:02.420(2)? ** on, your Meteor.publish() calls won't have much effect. All data
I20221103-03:15:02.421(2)? ** will still be sent to all clients.
I20221103-03:15:02.422(2)? **
I20221103-03:15:02.424(2)? ** Turn off autopublish by removing the autopublish package:
I20221103-03:15:02.426(2)? **
I20221103-03:15:02.427(2)? **   $ meteor remove autopublish
I20221103-03:15:02.428(2)? **
I20221103-03:15:02.429(2)? ** .. and make sure you have Meteor.publish() and Meteor.subscribe() calls
I20221103-03:15:02.429(2)? ** for each collection that you want clients to see.
I20221103-03:15:02.429(2)?
I20221103-03:15:02.478(2)? [FilesCollection] [find({}, undefined)]
I20221103-03:15:02.478(2)? [FilesCollection] [FilesCursor] [fetch()]
I20221103-03:15:02.481(2)? { files: [] }
I20221103-03:15:02.485(2)? []
I20221103-03:15:02.486(2)? [FilesCollection] [find({}, undefined)]
I20221103-03:15:02.487(2)? [FilesCollection] [FilesCursor] [fetch()]
I20221103-03:15:02.492(2)? { files: [] }
I20221103-03:15:02.492(2)? started
=> Meteor server restarted
I20221103-03:15:30.665(2)? [FilesCollection] [find({}, undefined)]
I20221103-03:46:02.374(2)? [FilesCollection] [find({}, undefined)]
I20221103-03:46:15.148(2)? [FilesCollection] [File Start Method] output3.wav - 896zbB9C9A3uw3Sd5
I20221103-03:46:15.150(2)? [FilesCollection] [Upload] [DDP Start Method] Got #-1/1 chunks, dst: output3.wav
I20221103-03:46:15.196(2)? Exception in callback of async function: Error: ENOENT: no such file or directory, mkdir
I20221103-03:46:15.196(2)?     at Object.mkdirSync (fs.js:1014:3)
I20221103-03:46:15.197(2)?     at packages/ostrio:files/write-stream.js:47:16
I20221103-03:46:15.199(2)?     at packages/ostrio:files/write-stream.js:10:50
I20221103-03:46:15.200(2)?     at runWithEnvironment (packages\meteor.js:1347:24) {
I20221103-03:46:15.201(2)?   errno: -4058,
I20221103-03:46:15.201(2)?   syscall: 'mkdir',
I20221103-03:46:15.202(2)?   code: 'ENOENT'
I20221103-03:46:15.202(2)? }
I20221103-03:46:15.215(2)? [FilesCollection] [Upload] [DDP] Got #1/1 chunks, dst: output3.wav
I20221103-03:46:15.220(2)? [FilesCollection] [Upload] [DDP] Got #-1/1 chunks, dst: output3.wav
I20221103-06:46:44.807(2)? [FilesCollection] [_preCollectionCursor.observe] [removed]: 896zbB9C9A3uw3Sd5
I20221103-06:46:44.809(2)? [FilesCollection] [_preCollectionCursor.observe] [removeUnfinishedUpload]: 896zbB9C9A3uw3Sd5
I20221103-06:46:44.818(2)? [FilesCollection] [Upload] [finish(ing)Upload] -> C:\AOC_tmp\wavs\896zbB9C9A3uw3Sd5.wav
I20221103-06:46:44.833(2)? [FilesCollection] [Upload] [finish(ed)Upload] -> C:\AOC_tmp\wavs\896zbB9C9A3uw3Sd5.wav
I

The storagePath was created (by the FilesCollections constructor, not by me) and it is empty:

C:\AOC_tmp\wavs>dir
 Volume in drive C is OS
 Volume Serial Number is 1A53-75A5

 Directory of C:\AOC_tmp\wavs

11/02/2022  09:45 PM    <DIR>          .
11/02/2022  09:45 PM    <DIR>          ..
               0 File(s)              0 bytes
               2 Dir(s)  11,220,852,736 bytes free
dr-dimitru commented 1 year ago

@Leekao please run same tests on v2.3.1, at least error would be more verbose

Leekao commented 1 year ago

@Leekao please run same tests on v2.3.1, at least error would be more verbose

This is the error I'm getting now:

I20221103-15:35:46.738(2)? Exception in callback of async function: errorClass [Error]: [FilesCollection] [writeStream] [constructor] [mkdirSync] ERROR: can not make/ensure directory  [500]
I20221103-15:35:46.739(2)?     at packages/ostrio:files/write-stream.js:50:21
I20221103-15:35:46.741(2)?     at packages/ostrio:files/write-stream.js:10:50
I20221103-15:35:46.742(2)?     at runWithEnvironment (packages\meteor.js:1347:24) {
I20221103-15:35:46.743(2)?   isClientSafe: true,
I20221103-15:35:46.744(2)?   error: 500,
I20221103-15:35:46.744(2)?   reason: '[FilesCollection] [writeStream] [constructor] [mkdirSync] ERROR: can not make/ensure directory ',
I20221103-15:35:46.745(2)?   details: Error: ENOENT: no such file or directory, mkdir
I20221103-15:35:46.745(2)?       at Object.mkdirSync (fs.js:1014:3)
I20221103-15:35:46.745(2)?       at packages/ostrio:files/write-stream.js:48:18
I20221103-15:35:46.747(2)?       at packages/ostrio:files/write-stream.js:10:50
I20221103-15:35:46.747(2)?       at runWithEnvironment (packages\meteor.js:1347:24) {
I20221103-15:35:46.749(2)?     errno: -4058,
I20221103-15:35:46.749(2)?     syscall: 'mkdir',
I20221103-15:35:46.750(2)?     code: 'ENOENT'
I20221103-15:35:46.750(2)?   },
I20221103-15:35:46.750(2)?   errorType: 'Meteor.Error'
I20221103-15:35:46.751(2)? }

I'm sorry but I still don't understand what is wrong.

dr-dimitru commented 1 year ago

@Leekao You can at least try

  1. Using other directories
  2. Using default directory (not setting any value at all)
dr-dimitru commented 1 year ago

I assume you pass empty directory from this error message:

[FilesCollection] [writeStream] [constructor] [mkdirSync] ERROR: can not make/ensure directory  [500]

See this line, message should end with the path you've retuned from your function, and it's empty string

dr-dimitru commented 1 year ago

@Leekao because you've never posted your actual code and FilesCollection definition.

Leekao commented 1 year ago

@Leekao You can at least try

  1. Using other directories
  2. Using default directory (not setting any value at all)

I tried that already and it did not work, this is my FilesCollection definition file:

import { Meteor } from 'meteor/meteor';
import { FilesCollection } from 'meteor/ostrio:files';
import path from 'path'

export const Wavs = new FilesCollection({
  collectionName: 'wavs',
  debug: true,
  storagePath: path.resolve('/AOC_tmp/wavs'),
});

if (Meteor.isClient) {
  Meteor.subscribe('files.wavs.all');
}

if (Meteor.isServer) {
  Meteor.publish('files.wavs.all', function () {
    return Wavs.find().cursor;
  });
}

And this is the code I use to insert:

 const onComplete = (blob) => {
    const outputFile = new File([blob], "output3.wav")
    console.log({outputFile})
    const upload_instance = Wavs.insert({
      file: outputFile,
      chunkSize: 'dynamic'
    }, true)
    upload_instance.on('end', (e, f) => {
      console.log(e, f)
      Meteor.call('New File', f)
    })
dr-dimitru commented 1 year ago

@Leekao try v2.3.2

Leekao commented 1 year ago

@Leekao try v2.3.2

Yes! It's working now! thank you very much for your patience!

dr-dimitru commented 1 year ago

Feel free to close it in case if the issue is solved on your end.

Leekao commented 1 year ago

@onurcansevinc is the issue fixed for you too?

onurcansevinc commented 1 year ago

@onurcansevinc is the issue fixed for you too?

I changed version, it solved. Thanks!

knoxgon commented 1 year ago

[SOLVED]

This issue still exists. The version bump to ostrio:files@2.3.2 introduced the problem. The previous version was @1.14.3 Intresting that it works for others.

And I haven't defined storagePath. Also, removed node_modules and local folder in .meteor to freshly install npm packages to ensure the removal of footprints of old packages.

Do I have to set storagePath to something? Previously, I dind't need to do that. The process has full access rights to the parent folder so I don't see any permission errors. The OS platform is Windows 10.

I read at the following post that the storagePath could be set to a defined space on disk but I don't really understand why this is required now. In such case, I think I could define it but I wonder what could've triggered for this change to be made. I'll be back after setting the storagePath.

export const UserFiles = new FilesCollection({
  collectionName: 'UserFiles',
  allowClientCode: false,
  schema: UserFilesSchema,
  onAfterUpload: ...,
  interceptDownload: ...,
  onAfterRemove: ...
});

EDIT: OK, I see that after defining the storagePath, it actually starts working. The process creates the named folder in C: disk as C:\.uploadedFiles but I don't find any files there. However, the files are pushed to the database.

export const UserFiles = new FilesCollection({
  collectionName: 'UserFiles',
  storagePath: '/.uploadedFiles/',
  allowClientCode: false,
  schema: UserFilesSchema,
  onAfterUpload: ...,
  interceptDownload: ...,
  onAfterRemove: ...
});