Private file server

[eMarek]

I am working on a project where users could upload photos. Uploading is made with CollectionFS and works perfectly. I would like to achieve that users could download their photos. Photos are private of course, so they could not be located in public folder. Further more each users photos will be packed in .zip file. Right know I am not sure yet how will a make it, but probably I will use somekind of a package which could .zip users photos. My question is, could CollectionFS serve this .zip file to particular user? How can I do it?

[raix]

I guess you could have a simple collection for the zip file generation - this could be done by attaching the user id as data in a file on a dummy SA as the "id" version of the file - now in the transformWrite you would be able to create the zip file and put it in the "zip" version of the "file"

The dummy storage adapter could be created as an empty storage adapter or a regular one.

User experience: The user clicks "generate zip" - the button goes waiting for the server - the id file is created and when the store "zip" is created the user would see the button go "download".

[eMarek]

This sounds perfect, but I am lost a little bit. So there should be a "generate zip" button which will call meteor method on the server. Than server have to generate zip file (from existing photos on disk which were uploaded using cfs:filesystem storage adapter) and save it somewhere on disk. After this zip file have to be put in a another CollectionFS, let's say zipFiles. Now server could return URL to zip file. User now have URL and could download his photos. Am I right? What is dummy storage adapter, how can I use it? Do you have maybe simple working demo somewhere?

[raix]

From a scaling point of view the zip functionality could be heavy duty - so at some point you should have the option to separate it from other parts of the app - From a user experience viewpoint it would be nice to see that the server is creating the zip version and prompting when ready.

You could also generate the zip when ever the user updates / uploads files - still queue-able / scaleable. This way it would be a simple link on the zip collection for the user to click (still they have to wait while the server generates the new zip) but a good user experience etc.

As you speak of a click and generate pattern, from my experience the user could end up waiting for the server to respond anyway (it has to generate and zip etc.) + the server would have no saying in when the zip should be generated / no way of flattening the load + what if one of the SA's fail while loading files to the zip SA? etc. its a bit harder to wait 10 minutes and try again.

A dummy SA is a SA that does nothing - so its just initializing the https://github.com/CollectionFS/Meteor-cfs-storage-adapter with noop functions as the api in https://github.com/CollectionFS/Meteor-cfs-storage-adapter/blob/master/storageAdapter.server.js#L8

Have a look at https://github.com/CollectionFS/Meteor-cfs-filesystem/blob/master/filesystem.server.js#L40

I think if you just create one SA "zipped" and pipe the zipped stream into the writestream (without using the readstream in the transformWrite) it should do the trick. You could remove the users zip file when the user uploads new file and when done add a file to the users zip collection - triggering the zip generation. (It would be nice to have some way of creating a fast gif animation of this supporting the text... please ask questions if I missed something)

So this example is completely untested, using the https://www.npmjs.org/package/archiver

  var files = new FS.Collection( ... );

  var archiver = require('archiver');

  var zippedFiles = new Collection( 'zipped', {
    stores: [
      new FS.Store.FileSystem('zip', {
       transformWrite: function(fileObj, readStream, writeStream) {
         // The readStream could contain the userId?
        var userId = '';
        readStream.on('data', function(chunk) {
          // XXX: we should limit the allowed data length, ending the stream if attacked?
          userId += chunk;
        });

        // When the user id is read we start the zip stuff..
        readStream.on('end', function() {
          if (userId) {
            // zipper
            var archive = archiver('zip');

            // Rig the write piping
            archive.pipe(writeStream);

            // You have to get readstreams from files. Something like
            files.find({ owner: userId }).forEach(function (fileObj) {
              var fileReadStream = fileObj.createReadStream('images');
              // Do some stuff... like
              archive.append(fileReadStream, fileObj.name);
            });

            // XXX: This is just copied from the node-archiver example. Haven read up on this...
            // But you have to call finalize when all files are added to archive.
            archive.finalize();
          }
        });
       }
     })
    ]
  }):

Something like this would be a dummy SA - but I dont think you need it the above describes a better pattern. Server:

  FS.Store.Dummy = function(name, options) {
    var self = this;
    if (!(self instanceof FS.Store.Dummy))
      throw new Error('FS.Store.Dummy missing keyword "new"');

    options = options || {};

    return new FS.StorageAdapter(name, options, {
      typeName: 'storage.dummy',
      fileKey: function(fileObj) {
        return fileObj.collectionName + '-' + fileObj._id;
      },
      createReadStream: FS.Utility.noop,
      createWriteStream: FS.Utility.noop,
      remove: FS.Utility.noop,
      // stats: FS.Utility.noop // optional - not used in cfs yet
    });
  };

Client:

// On the client we have just a shell
FS.Store.Dummy = function(name, options) {
  var self = this;
  if (!(self instanceof FS.Store.Dummy))
    throw new Error('FS.Store.Dummy missing keyword "new"');

  return new FS.StorageAdapter(name, options, {
    typeName: 'storage.dummy'
  });
};

In case it proves useful I could add a package repo cfs-sa-dummy and add you on the team?

[eMarek]

Thank you for your answer. It helped me a lot. So this is my version. After user uploads a photo, I make 10 seconds time out, and than a zip file is created which is stored in Collection FS. Something like this:

Meteor.setTimeout(function() {
    // find all photos with the same "waterfallId"
    var waterfallPhotos = Flow.find({
        $and: [
            {waterfall: waterfallId},
            {type: "photo"}
        ]
    }, {
        fields: {
            'photo': 1
        }
    }).fetch();

    // find all photos in CollectionFS
    var collectionPhotos = Photos.find({
        _id: {$in: _.pluck(waterfallPhotos, 'photo')}
    }, {
        sort: {uploadedAt: 1}
    }).fetch();

    // prepare data for zipping
    var fs = Npm.require('fs');
    var zip = new JSZip();
    var dir = process.env.PWD;
    var photosDir = dir + "/uploads/photos-300x300/";
    var temporaryDir = dir + "/tmp/";

    // zip each photo
    _.each(collectionPhotos, function(photo, i) {
        try {
            zip.file("photo-" + (("000" + (i + 1)).slice(-3)) + "-" + photo.copies.photo.name, fs.readFileSync(photosDir + photo.copies.photo.key));
        } catch (e) {
            console.log(e);
        }
    });

    // save zip file in temporary folder
    var temporaryZip = temporaryDir + "photos-" + new Date().getTime() + ".zip";
    zip.saveAs(temporaryZip);

    // save zip file in CollectionFS
    var zipFile = new FS.File(temporaryZip);
    zipFile.metadata = {
      waterfallId: waterfallId
    };
    Zipped.insert(zipFile);

}, 10000);

I still have some work. Removing temporary zip files on disk and removing older zip files in CollectionFS. Basically it is a little bit odd because I make two copies of zip file anytime a new photo is uploaded. Do you maybe know how could I get rid of it? Maybe with this dummy storage adapter? I really don't know how to use it. I don't have enough knowledge.