Uploading frequently fails in production

[andrroy]

Getting this error in production:

Error: Error storing uploaded file to TempStore: no open connections
    at EventEmitter.<anonymous> (packages/cfs_collection/packages/cfs_collection.js:161:1)
    at EventEmitter.emit (events.js:98:17)
    at Stream.<anonymous> (packages/cfs_tempstore/packages/cfs_tempstore.js:343:1)
    at Stream.emit (events.js:117:20)
    at Stream._error (/mnt/app/programs/server/npm/cfs_gridfs/node_modules/gridfs-stream/lib/writestream.js:190:8)
    at /mnt/app/programs/server/npm/cfs_gridfs/node_modules/gridfs-stream/lib/writestream.js:175:26
    at error (/mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:274:5)
    at /mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:178:22
    at /mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/collection/query.js:163:28
    at commandHandler (/mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:707:48)

The error is usually triggered when more than one file is uploaded in parallell. The app is currently running on Modulus.io. I have not been able to recreate the problem locally in my dev environment. All files uploaded are less than 1MB. Any suggestions on what might be the problem?

[andrroy]

Update: The frequency of the problem dropped after we removed the callback from inserts. But I am still experiencing the exact same problem. 10 inserts failed earlier today, with the following error message:

fileObj.createReadStream creating read stream for attached data
createWriteStream _tempstore, internal: true
createWriteStreamForFileKey _tempstore
-----------CLOSE STREAM _tempstore
SA GridFS - ERROR! [Error: no open connections]
-----------ERROR STREAM _tempstore no open connections
TempStore writeStream error: [Error: no open connections]

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: Error storing uploaded file to TempStore: no open connections
    at EventEmitter.<anonymous> (packages/cfs_collection/packages/cfs_collection.js:161:1)
    at EventEmitter.emit (events.js:98:17)
    at Stream.<anonymous> (packages/cfs_tempstore/packages/cfs_tempstore.js:343:1)
    at Stream.emit (events.js:117:20)
    at Stream._error (/mnt/app/programs/server/npm/cfs_gridfs/node_modules/gridfs-stream/lib/writestream.js:190:8)
    at /mnt/app/programs/server/npm/cfs_gridfs/node_modules/gridfs-stream/lib/writestream.js:175:26
    at error (/mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:274:5)
    at /mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:178:22
    at /mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/collection/query.js:163:28
    at commandHandler (/mnt/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:707:48)

All 10 files are in the cfs._tempstore.chunks collection:

{ _id: "kLXs48Bpzb35C8xDN", fileId: "7hRfFi7HtjJpgjgpX", collectionName: "Files", keys: {} }
{ _id: "8bs9ShHELbjqn7jHa", fileId: "ovMR5BRh2wbRGTbad", collectionName: "Files", keys: {} }
{ _id: "nKkjNKbmGKMwr8qfj", fileId: "DERK7H5HKdLM4HzqL", collectionName: "Files", keys: {} }
{ _id: "e86m6qk3LDpDCkXCK", fileId: "pupWYttH4JmhBpRgL", collectionName: "Files", keys: {} }
{ _id: "bvPjA3QSHZa7z9xk9", fileId: "W6CSo3xY7acDnFyDZ", collectionName: "Files", keys: {} }
{ _id: "odGAxSzTmjf3Nh7fE", fileId: "cMnZmLycnFvy9GNeh", collectionName: "Files", keys: {} }
{ _id: "kdwxjSq6m6quC2476", fileId: "Bm8XkmQmDARSppiyJ", collectionName: "Files", keys: {} }
{ _id: "RjptdQwMwEiuM78nm", fileId: "2xNabHjjJQQZf2Eoh", collectionName: "Files", keys: {} }
{ _id: "7DMQ7kWawtQHhR2ej", fileId: "REyB6cfAHQoNjLxoF", collectionName: "Files", keys: {} }
{ _id: "SM5khtQbBAwbSLndx", fileId: "RKbGg4z8yXYpbiJvj", collectionName: "Files", keys: {} }

[digz6666]

Confirmed, uploading fails sometimes when multiple files uploaded in parallel. It throws following error and meteor server restarts. It says permission error which is misleading. Folder upload permission is configured correctly.

Error: EACCES, open '/opt/autoland/uploads/thumbs/images-3pQemi6x6mPMopxuf-1.jpg'
error: Forever detected script exited with code: 7
error: Script restart attempt #70

By the way do you have cfs:gridfs package added to your meteor app?

[andrroy]

I have the cfs:gridfs package installed, yes. We might have found a "fix" to the issue btw: As previously mentioned, the frequency of the problem dropped after removing callbacks from inserts. We also had callbacks for other FS-functions. After removing callbacks from all FS-calls, the problem seems to have stopped.

I have no idea why this works - I'm just happy it does!

[digz6666]

@andrroy Thanks for your suggestion. You mean insert callback on client side? I'm inserting image and saving its reference to another collection. How can I get the id of the inserted image without using callback?

var carId = ...

var newFile = new FS.File();
newFile.carId = carId;
newFile.isCover = true;
newFile.attachData(file, {type: 'image/jpg'}, function(error) {
    if(error) {
        throw error;
    }
    newFile.name('cover.jpg');

    Images.insert(newFile, function(err, fileObj) {
        if(err) {
            throw err;
        }
        Car.update({ _id: carId }, { $set: {coverImage: fileObj} });
    });
});

[andrroy]

@digz6666 Try this:

var carId = ...

var newFile = new FS.File();
newFile.carId = carId;
newFile.isCover = true;
newFile.attachData(file, {type: 'image/jpg'});
newFile.name('cover.jpg');

var file_id = Images.insert(newFile);
Car.update({ _id: carId }, { $set: {coverImage: file_id} });

This will only work on the server-side btw!

[snajjar]

Hello,

We are observing the same issue (using cfs:gridfs).

SA GridFS - ERROR! [Error: no open connections]
^@^@^@^@
events.js:72
     throw er; // Unhandled 'error' event
        ^
Error: Error storing uploaded file to TempStore: no open connections
  at EventEmitter.<anonymous> (packages/cfs_collection/packages/cfs_collection.js:161:1)
  at EventEmitter.emit (events.js:117:20)
  at Stream.<anonymous> (packages/cfs_tempstore/packages/cfs_tempstore.js:343:1)
  at Stream.emit (events.js:117:20)
  at Stream._error (/home/vcap/app/.bluemix/app/programs/server/npm/cfs_gridfs/node_modules/gridfs-stream/lib/writestream.js:190:8)
  at /home/vcap/app/.bluemix/app/programs/server/npm/cfs_gridfs/node_modules/gridfs-stream/lib/writestream.js:175:26
  at error (/home/vcap/app/.bluemix/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:274:5)
  at /home/vcap/app/.bluemix/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:178:22
  at /home/vcap/app/.bluemix/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/collection/query.js:163:28
  at commandHandler (/home/vcap/app/.bluemix/app/programs/server/npm/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:707:48)

Our app is running on Bluemix, database on Compose. We're also using callback on insertions/update, but it seems really heavy to remove them all...

How can we advance on this one?

[andrroy]

@raix, @aldeed Is this project still maintained?

[snajjar]

@andrroy: i wonder if the issue is related to the usage of MongoDB 3.2. Mine is 3.2, what version do you use? See https://github.com/meteor/meteor/issues/5809

btw: It seems that this package is in a "unclear" situation about maintenance. See https://forums.meteor.com/t/deprecating-collectionfs/16921

[andrroy]

@snajjar: Nice find! We are using 3.2 as well.

We are currently working on migrating away from this project. CollectionFS is only used because of the S3 functionality. Because of this, we are currently working on porting our app over to peerlibrary:aws-sdk.

[VansonLeung]

I would test MongoDB 3.2 compatibility with out Meteor CMS project now. We heavily rely on GridFS to upload files and images (and we use autoform-files) with all our projects and we need to have relational $lookup support for searching. I love the fact MongoDB 3.2 supports primitive joins which means we do not need to reconsider using MySQL anymore.