vsivsi / meteor-file-collection

Extends Meteor Collections to handle file data using MongoDB gridFS.
http://atmospherejs.com/vsivsi/file-collection
Other
159 stars 37 forks source link

Migrating files from another system #70

Closed dpatte closed 8 years ago

dpatte commented 8 years ago

Thank you for your package. It is working great for me.

But I now have a few questions about a file migration that I am doing from another system into my app. I have hundreds of files including .doc and .xls files on another system that I would like to migrate into file collection in Meteor.

My thought was to sftp all the files into the app's private folder on my running deployed server then run a special migration function using fc.importFile to load them into my file collection. Initial tests on my dev system look promising, but I have a few questions, if you don't mind...

1) The filepath on my dev system is, for example: /home/dpatte/myapp/private/file1.xls. Using this full path in fc.importFile works great on my dev system, but when deployed the path will be different. Are there any recommendations as to where the files should be placed for import (does it matter?), and how would you recommend that I handle the fact that the path is different between my dev and deployed systems? The files will be sftp'd after the app is deployed.

2) In other places in my system where I use fc.Insert, I am using resumable, and it provides me with the contentType in file.file.type. What do you recommend I use to determine the filetype when using importFile?

3) If an expected file doesnt exist at path, a STDERR Error: ENOENT is thrown and the fc.importFile callback doesn't seem to see it. I see the server console message, the system rebuilds and goes into a rethrowing loop. I expected that the importFile callback would present the error to me to handle. How would I simply exit the current method in this case?

4) Finally, the initial comment in your sample for importFile is incorrect.

Thanks for your help.

vsivsi commented 8 years ago

Hi, thanks for your questions.

For 1) It probably depends what your deployment host allows. I know that some services have restrictions on access to the local filesystem (paths and capacity), so you'll be forced to figure out where and how much data you are allowed to write for import.

On 2): when you use resumable, it is determining the filetype based on the browser's mapping of file extensions to mime types. With 15 seconds of searching on npm, this node.js package seems to be wildly popular: https://www.npmjs.com/package/mime-types. On atmosphere there appears to be a meteor package based on a different npm package here: https://atmospherejs.com/patte/mime-npm

3) appears to be a bug in the createReadStream here: https://github.com/vsivsi/meteor-file-collection/blob/master/src/gridFS_server.coffee#L324 I'll fix that in the next release.

4) Thanks for noting.

vsivsi commented 8 years ago

One other note. fc.importFile() is meant as convenience function. And its implementation is extremely simple: https://github.com/vsivsi/meteor-file-collection/blob/master/src/gridFS_server.coffee#L319-L328

If you find that you need more specialized functionality, I wouldn't hesitate to just grab that code into your app and hack away at it until it does what you need it to.

dpatte commented 8 years ago

1) I notice if I deploy the files in assets, they are accessible. I have yet to try to sftp them - I suspect I may end up with owner issues. 2) Thanks for this. Nice coincidence as well, considering the developer's handle (patte), and mine (dpatte). :) 3) Will keep my eyes open for that. 4) No problem :)

vsivsi commented 8 years ago

Fixed in version 1.2.2 on Atmosphere now.

dpatte commented 8 years ago

Great. Will give it a try and let you know how it goes.

vsivsi commented 8 years ago

Great, thanks.

dpatte commented 8 years ago

I worked around this a different way, importing the files manually.