Avoid files being deleted on meteor deploy

[cmeethree]

It seems whenever you deploy any changes it wipes out the files in the cfs filesystem such that any images that a user had saved into that file system get wiped out.

Is there some way so that when you deploy it doesn't overwrite the files in cfs/files that currently exist?

Example: user creates post and image is stored at /cfs/files/images/czTTzk7AZFPNDti2S

I re-deploy application using meteor deploy after making a change to something in the application.

The image/file system at /cfs/files/images/czTTzk7AZFPNDti2S seems to be overwritten upon deploy such that the image no longer shows up.

Thanks.

[nooitaf]

You might want to concider using https://github.com/CollectionFS/Meteor-cfs-gridfs instead of filesystem, because gridfs stores your files in the MongoDB and not on disc. And the DB stays the same after deploy, if i recall correctly.

[cmeethree]

Ok great I'll try that.

On Fri, Oct 24, 2014 at 1:45 PM, Ralf Brandenstein <notifications@github.com

wrote:

You might want to concider using https://github.com/CollectionFS/Meteor-cfs-gridfs instead of filesystem, because gridfs stores your files in the MongoDB and not on disc. And the DB stays the same after deploy, if i recall correctly.

— Reply to this email directly or view it on GitHub https://github.com/CollectionFS/Meteor-CollectionFS/issues/460#issuecomment-60423288 .

twtr: @cmeethree tmblr: http://cmeethree.tumblr.com/ linkdin: www.linkedin.com/in/chazmee/

[aldeed]

Yeah, meteor deploy doesn't have persistent filesystem but the default DB should be persistent I think or you can point it to your own MongoDB.

[nooitaf]

Just tested it, you can leave out any options and it will be persistent.

var ImagesStore = new FS.Store.GridFS('images-original');

https://github.com/nooitaf/cfs-imagetest-094

[cmeethree]

Ok cool. Btw I notice you use "image-original" - what's the purpose between using this vs "image".

Sent from my iPhone

On Oct 25, 2014, at 2:25 PM, Ralf Brandenstein notifications@github.com wrote:

Just tested it, you can leave out any options and it will be persistent.

var ImagesStore = new FS.Store.GridFS('images-original'); https://github.com/nooitaf/cfs-imagetest-094

— Reply to this email directly or view it on GitHub.

[nooitaf]

It's the name of the Store. It's easier to remember that those are unmanipulated images. Usually i have more Stores...

images-original
images-1024
images-256
images-64

"Original" is the original file uploaded and the others are smaller versions/thumbnails scaled down by gm/im. Then, when i use the url helper {{url store="images-64"}} i get the image with 64 pixel width. Meteor servers don't have (didn't test recently) imagemagick/graphicsmagick installed, so you cant manipulate images on their servers. But if you host it yourself it's less of a problem.

You could also replace images- with imagestore- or store- or just files-original. It might be better to say thumbs-256 but i settled with the images- prefix.

Also it's easier to have distinct names, instead of a collection called images and a store called images and a variable called images and maybe even another mongo collection called images. All unrelated but with the same name. Can get very confusing :\

[nooitaf]

I should have given the variable a fitting name too.

var ImagesOriginalStore = new FS.Store.GridFS('images-original');

[cmeethree]

Interesting. Yeah I'm not using GridFS but the file store.

Was trying to figure out how to deploy without it wiping out the image folder everytime. I'll check this out - have a few higher priority items to fix that have a higher friction rating for me.

On Oct 25, 2014, at 7:17 PM, Ralf Brandenstein notifications@github.com wrote:

I should have given the variable a fitting name too.

var ImagesOriginalStore = new FS.Store.GridFS('images-original'); — Reply to this email directly or view it on GitHub.

[cmeethree]

Hi Ralf (been a busy week). But this worked perfectly. I'm getting more of a hang of CollectionFS. Switching to GridFS also appeared to improve some performance stuff as well. Don't know if that's actually true or not.

/c

On Sun, Oct 26, 2014 at 11:51 AM, Chaz Mee chaz.mee@gmail.com wrote:

Interesting. Yeah I'm not using GridFS but the file store.

Was trying to figure out how to deploy without it wiping out the image folder everytime. I'll check this out - have a few higher priority items to fix that have a higher friction rating for me.

On Oct 25, 2014, at 7:17 PM, Ralf Brandenstein notifications@github.com wrote:

I should have given the variable a fitting name too.

var ImagesOriginalStore = new FS.Store.GridFS('images-original');

— Reply to this email directly or view it on GitHub https://github.com/CollectionFS/Meteor-CollectionFS/issues/460#issuecomment-60500820 .

twtr: @cmeethree tmblr: http://cmeethree.tumblr.com/ linkdin: www.linkedin.com/in/chazmee/

[Ethaan]

Just asking

i have a full app alredy done, but im using cfs:fileSystem, the app works fine (for us purpose), but now im afraid when im deploy that app to heroku or whatever host service, all my file get deleted, i have several questions about this

Another question, everytime i make a change on the meteor app (now talking that the app its alredy on some hostServer, the existing files will get deleted?), all the time? i should stop using cfs:filesystem? its the same between cfs:filesystem or gridfs? or is a radical change between that 2 storage adapters?

[aldeed]

It depends on the hosting provider. Modulus provides a persistent cloud storage path, so you could use that. But in general, GridFS or s3 are better if you're not hosting on your own server. GridFS stores to the database, so as long as your MONGO_URL stays the same, the files are still there. However, database space is usually more expensive than disk space, so using S3 can be more economical.

[Ethaan]

@aldeed man thank you, you know me and my mates are very noobies on meteor, and alredy have 95% of an application made with cfs:filesystem

thanks for all the support

[cmeethree]

Hey total beginner here - I built my first real app using CollectionFS in Sept/Oct - started with the filesystem and ran into the same thing (deleting of images on deploy), but switched to GridFS and it works great. Didn't actually have too much of an effect my switching over - it's not that radical a change. But I would switch over.

On Tue, Nov 4, 2014 at 8:43 PM, Ethaan notifications@github.com wrote:

@aldeed https://github.com/aldeed man thank you, you know me and my mates are very noobies on meteor, and alredy have 95% of an application made with cfs:filesystem

thanks for all the support

— Reply to this email directly or view it on GitHub https://github.com/CollectionFS/Meteor-CollectionFS/issues/460#issuecomment-61747167 .

Join me on my latest project called Placelist - Placelist. Keep a list of the places you love to go. http://placelist.co/

twtr: @cmeethree tmblr: http://cmeethree.tumblr.com/ linkdin: www.linkedin.com/in/chazmee/

[robmagario]

Is there any way to deploy using filesystem instead of GridFS and still keeping the images? I noticed that every time I deploy, the images get deleted if I use filesystem. You guys suggested GridFS. I was using GridFS before and yes, the images don't get deleted after deploying, because they are saved in the database. Having that said, I also noticed that my site became much slower than before because those images need to be read from the database now and they cannot be cached since GridFS doesn't cache those images.

[cmeethree]

This seems like an old email. I’m using this and the files don’t get deleted. The images are not saved in the database - rather it seems a pointer to the image is saved in the DB. I figured this out after playing around for a while. I actually don’t have performance issues.

/c

[nooitaf]

@robmagario i suggested gridfs if you want to deploy on meteor.com. If you deploy to your own server you can ofc use filesystem.

[Sharlaan]

I'm using cfs:gridfs & cfs:graphicsmagick, but :

1. cfs:graphicsmagick path seems to not be recognized. i read on some article there might have an error with the regex used in path detection code for Windows, which has been fixed in original github code but not the meteor package.... so iended up installing it through meteorhacks:npm + NPMrequire("gm") Any way to fix the meteor package cfs:graphicsmagick instead of using this hacky way ?
2. on meteor deploy, my page runs ok but the upload fails always (infinite loop on the PUT request) i suspected this to be related to GM not being able to install on .meteor.com servers, right ? If my suspicions are correct, is there any way to make a thumbnail system working with .meteor.com ?

[nooitaf]

_2. As there is no Graphicsmagick installed on the meteor servers you cant use it ofc. You could try to create a thumbnail on the client by parsing it through a canvas and upload 2 files. Or maybe use some server side canvas package. (https://github.com/jagi/meteor-node-canvas/ for example (there might be better packages))

[Sharlaan]

Ok thanks, guess i'll drop gm/im option for thumbnail since can't deploy them... Just wondering how to upload to a specific Store ? i just have (clientside): MyImagesFScollection.insert(fsFile, callback) (serverside) MyImagesFScollection = new FS.Collection("images", { stores: [ new FS.Store.GridFS("thumbsStore"), new FS.Store.GridFS("imagesStore") ] is there a way to to force the insert with store="thumbsStore" so the server knows to NOT store in store="imagesStore" ? like for example: (clientside) MyImagesFScollection.insert(fsFileThumb, {store: "thumbsStore"} callback) hoping to not have to create a collection specifically just for thumbs ? sounds complicated to retrieve them after in template ?

[nooitaf]

Well, you could use 2 collections and then insert them individualy

[jeffyamada]

Hey guys,

I'm pushing and pulling copies of my app's db between local development machines, development servers, and staging etc. I noticed whenever I write-over an environment's db, the files not referenced in the updated copy get deleted (on S3). This is problematic in a number of scenarios that I'm sure you can quickly imagine (someone simply resets their local and then POOF....).

I'm hunting around as I suspect there might be some simple solutions for this - but I've so far had limited success altering the S3 delete permissions and so I'm hoping to prevent deletion (and then ensure unique naming of new files) etc.

If anyone has faced this issue and has a solution already, or knows what I'm doing wrong many thanks!

Cheers

[jeffyamada]

Ah, FS.Worker removes files that aren't in a collection - so I've updated my migration scripts to use --excludeCollection=cfs.[your-collection].filerecord which will preserve local collections. This essentially means that all environments will share the files and the collection, so if anyone else runs into this just be sure to write scripts to manage production/non-production files.