Remove does not always work

[ignl]

Hi, after I started testing with bigger video files I noticed that remove sometimes does not work. As I understand it has something to do with locking. Once video starts playing and I try to remove it sometimes it works immediately and sometimes doesn't work at all. It invokes remove() method but callback break point is not hit when debugging. Is there a way to not use locking at all? Files cannot be updated in my application (maybe only metadata and I don't care that much about about consistency - that other all users must see the same information). However when user wants to delete a file - it must be deleted immediately. How can achieve that?

[ignl]

I also noticed a lot of these in production maybe it's related?

(node) warning: possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit. App 19778 stderr: Trace App 19778 stderr: at EventEmitter.addListener (events.js:239:17) App 19778 stderr: at EventEmitter.once (events.js:265:8) App 19778 stderr: at GridReadStream.stream.renewLock (/var/www/vdling/bundle/programs/server/npm/node_modules/meteor/vsivsi_file-collection/node_modules/gridfs-locking-stream/index.js:188:18) App 19778 stderr: at GridReadStream. (packages/vsivsi_file-collection/src/gridFS_server.coffee:300:33) App 19778 stderr: at emitNone (events.js:67:13) App 19778 stderr: at GridReadStream.emit (events.js:166:7) App 19778 stderr: at EventEmitter. (/var/www/myapp/bundle/programs/server/npm/node_modules/meteor/vsivsi_file-collection/node_modules/gridfs-locking-stream/index.js:205:54) App 19778 stderr: at emitOne (events.js:77:13) App 19778 stderr: at EventEmitter.emit (events.js:169:7)

Meteor 1.4.1.1

[vsivsi]

Hi, sorry for the delay in responding, I've been on vacation for a couple of weeks. Yes, it sounds like you are running up against the underlying file locking system.

Probably the easiest thing to try is to set the lockExpiration and pollingInterval options to be 1 (one second each). That will allow a file to be removed within a second of the beginning of being read. If a file has many readers, then obtaining the write lock may take a little longer, and in the worst case (a continuous stream of new readers, faster than one per second) removal would still be blocked. Only you can determine if that scenario is likely or subject to abuse in your application.

It is also possible to directly release the read lock for a given stream (returned by say findOneStream()) by calling its stream.releaseLock([callback]) method. However, if you are using the built-in HTTP support, you won't typically have direct access to this stream object for a given connection, and of course if there are multiple readers, you would need to unlock each of them.

You could also manually unlock a file by directly manipulating the lock collection, but I wouldn't recommend that unless you really, really need absolute control over this. If you take that path, you would need to work very hard to avoid race conditions on busy files.

If none of the above work for you, it's possible that I could add a "force" option to fc.remove() that would always remove a file regardless of its lock status. It would involve circumventing the locking system, as described above, but probably wouldn't be too difficult to implement.

[ignl]

Hi, no problem. Yes I saw those options in documentation. However setting lockExpirationand pollingIntervaldidn't help - somehow video is still not removed even though only I watch it and wait for a bit before trying to delete it. Force option would be definitely good addition. On the other hand my use case is to upload file (video/image) and then other users can see but file itself it is never modified after that. It only can be removed and remove operation must be immediate (so it's ok if someone watched a video and suddenly it got deleted). Maybe locking is not needed at all?

[edemaine]

I'm also getting the "EventEmitter memory leak detected" messages in deployment. Is this caused by too many accesses to the same file or something?

[vsivsi]

@edemaine Are you running the latest version? This was fixed in 1.3.7

[vsivsi]

Closing this, as it seems resolved.