Files 'disappearing'?

[elewis33]

This is a question, and may not be an issue. I looked in the current issues and didn't see anything that stood out as the same/similar to what I'm asking.

I'm using Meteor-Files to download some zip files from a web site and then store them for processing in my application. Here is my Meteor.Files configuration:

this.Uploads = new Meteor.Files({
  debug: true,
  collectionName: 'Uploads',
  allowClientCode: false, // Disallow remove files from Client
  public: true,
  downloadRoute: 'public/surveys/wireless/zips/',
  storagePath: 'assets/app/zips/',
  onBeforeUpload: function (file) {
    // control file size and allowed file formats here
    if (file.size <= 1024*1024*20 && /zip|png|jpg|jpeg|mp3|mp4|mpeg/i.test(file.extension)) {
      return true;
    } else {
      return 'Please limit upload size to 20MB';
    }
  }
});

My app is working great. I have a separate collection that tells me where the zip files are located, by URI, so I use the load() method to grab the files and store them locally and all seems to be awesome. However, after I shutdown my app and come back to it the files that I downloaded disappear. I look in the storagePath when the app is running and everything looks great. After I shutdown/restart meteor they're all gone. Am I missing something? Do I need to do something differently to persist the files that I'm downloading/storing?

[yanickrochon]

Same thing.

Where files is stored by default?: by default if config.storagePath isn't passed into Constructor it's equals to assets/app/uploads and relative to running script:

• On development stage: yourDevAppDir/.meteor/local/build/programs/server, note: all files will be removed as soon as your application will rebuild or you run meteor reset. To keep you storage persistent use absolute path outside of your project folder, we recommend to use /data directory
• On production: yourProdAppDir/programs/server

Sure, that's what I do!

const Collection = new FilesCollection({
    collectionName: 'Images',
    storagePath: 'data/images',
    parentDirPermissions: 0774,
    permissions: 0774,
    allowClientCode: false,
    debug: Meteor.isServer && process.env.NODE_ENV === 'development',
});

Except that, even if I specify data, the files are still written in ./.meteor/local/build/programs/server/data/images/ and are deleted when the app restarts (i.e. is rebuilt).

I have tried setting ./data/images/ but the behavior is unchanged.

[dr-dimitru]

@elewis33 and @yanickrochon

Exactly as it said in docs: all files will be removed as soon as your application will rebuild!.

To avoid this behaviour use absolute path to your FileSystem. And outside of the Meteor project directory

To do so - start your path in storagePath with / (slash)

Do not forget to set proper permissions

[elewis33]

Love the Walter meme! Thanks for answering. I was reading the "full API" documentation and it says nothing about this. Never actually saw that comment in the FAQ, so I guess I learned that I need to read all of the documentation. Do you want me to do a PR to fix the documentation here: https://github.com/VeliovGroup/Meteor-Files/wiki/Constructor? config.storagePath says nothing about this rule on that page. And that is supposed to be the "Full API" documentation.

[elewis33]

Also, if you're going to remove the files as stated in the FAQ don't you think it would be a good idea to truncate the Mongo collection too, so things stay in sync? The fact that db.Uploads.find().count() was returning something (not zero) and the files were gone was a little perplexing.

[dr-dimitru]

Love the Walter meme! ... documentation and it says nothing about this.

:) This was reaction to path difference with and without leading /

Never actually saw that comment. ... Do you want me to do a PR

Yes, PRs is always welcome

if you're going to remove the files as stated in the FAQ..

Actually, not I am nor MF-lib removing the files. The Meteor removes them upon rebuilding

.. truncate the Mongo collection too, so things stay in sync?

Can't do it as Meteor may change this behaviour (the way it rebuilds), then some other things may go wrong

[dr-dimitru]

@elewis33 glad you have solved it. Waiting for PR - to make docs better.

Please, support this project by:

• Star on GitHub
• Star on Atmosphere
• Share on Facebook and Twitter

[yanickrochon]

@dr-dimitru the fact that Meteor treats a leading slash as "the root of the project" and not the root of the FS makes it absurd for you to call "rules" when what you define is either improperly documented and inconsistent with Meteor's behavior.

Don't get mad because people aren't in your head to understand what you mean. Especially since /data is not necessarily a good choice to store data in the file system; for example, /var/data would've been a better suggestion.

[dr-dimitru]

@yanickrochon

I'm saying what speaking of file system - leading slash always means root of your machine's file system, in any programming language on any platform.

Yes, Meteor has it's own root for so-called Assets, that's it. In all other cases it's node.js and JavaScript, where FS root is your machine's file system root.

/data is given as example with leading slash, you're free to use any directory you're comfortable with.

About docs - it may be not perfect, if you're not satisfied with it - send PR, make it better.

[anaghakomawar]

I am trying to use Meter Files. I want to store my files on S3 bucket.

Problem which I am facing is : I am able to store video on directory, later it seams that it is uploading the file on S3 (which is not working) as it is not getting any error it is unlinking the video. So it deletes the video from the directory. This way I am loosing the data. Following is my code:

Server/videoupload.js

import { Meteor } from 'meteor/meteor'; import { _ } from 'meteor/underscore'; import { Random } from 'meteor/random'; import { FilesCollection } from 'meteor/ostrio:files'; import stream from 'stream';

import S3 from 'aws-sdk/clients/s3'; // http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html // See fs-extra and graceful-fs NPM packages // For better i/o performance import fs from 'fs';

process.env.S3='{"s3":{"key": "xxx", "secret": "xxx", "bucket": "xxx", "region": "xxx""}}'

if (process.env.S3) { Meteor.settings.s3 = JSON.parse(process.env.S3).s3; } // console.log(Meteor.settings.s3); const s3Conf = Meteor.settings.s3 || {}; const bound = Meteor.bindEnvironment((callback) => { return callback(); });

// Check settings existence in Meteor.settings // This is the best practice for app security if (s3Conf && s3Conf.key && s3Conf.secret && s3Conf.bucket && s3Conf.region) { // Create a new S3 object //s3Conf.secret, //s3Conf.key, //s3Conf.region, const s3 = new S3({ secretAccessKey : s3Conf.secret, accessKeyId : s3Conf.key,
region : s3Conf.region,
// sslEnabled: true, // optional httpOptions: { timeout : 60000, agent : false } });

// console.log('s3: ', s3);
// Declare the Meteor file collection on the Server
export const BizVideo = new FilesCollection({
    debug: true, // Change to `true` for debugging
    storagePath: 'bizVideos',
    collectionName: 'bussVideo',
    // Disallow Client to execute remove, use the Meteor.method
    allowClientCode: false,
    chunkSize: 1024 * 1024,

    // Start moving files to AWS:S3
    // after fully received by the Meteor server
    onAfterUpload(fileRef) {
        // Run through each of the uploaded file
        // console.log("fileRef2: ", fileRef);
        _.each(fileRef.versions, (vRef, version) => {
            // We use Random.id() instead of real file's _id
            // to secure files from reverse engineering on the AWS client
            const filePath = 'files/' + (Random.id()) + '-' + version + '.' + fileRef.extension;
            console.log("filePath: ", filePath);

            // Create the AWS:S3 object.
            // Feel free to change the storage class from, see the documentation,
            // `STANDARD_IA` is the best deal for low access files.
            // Key is the file name we are creating on AWS:S3, so it will be like files/XXXXXXXXXXXXXXXXX-original.XXXX
            // Body is the file stream we are sending to AWS
            s3.putObject({
                // ServerSideEncryption: 'AES256', // Optional
                StorageClass : 'STANDARD_IA',
                Bucket       : s3Conf.bucket,         //s3Conf.bucket,
                Key          : filePath,
                Body         : fs.createReadStream(vRef.path),
                ContentType  : vRef.type,
            }, (error) => {
                // console.log("error: ", error);
                bound(() => {
                    if (error) {
                        console.error(error);
                    } else {
                        // Update FilesCollection with link to the file at AWS
                        const upd = { $set: {} };
                        upd['$set']['versions.' + version + '.meta.pipePath'] = filePath;
                        console.log("upd: ", upd);

                        this.collection.update({
                            _id: fileRef._id
                        }, upd, (updError) => {
                            if (updError) {
                                // console.log("updError: ", updError);
                                console.error(updError);
                            } else {
                                // Unlink original files from FS after successful upload to AWS:S3
                                console.log("unlink: ", fileRef._id);
                                this.unlink(this.collection.findOne(fileRef._id), version);
                            }
                        });
                    }
                });
            });
        });
    },

    // Intercept access to the file
    // And redirect request to AWS:S3
    interceptDownload(http, fileRef, version) {
        console.log('interceptDownload');
        let path;

        if (fileRef && fileRef.versions && fileRef.versions[version] && fileRef.versions[version].meta && fileRef.versions[version].meta.pipePath) {
            path = fileRef.versions[version].meta.pipePath;
        }

        if (path) {
            console.log('path ',path);
            // If file is successfully moved to AWS:S3
            // We will pipe request to AWS:S3
            // So, original link will stay always secure

            // To force ?play and ?download parameters
            // and to keep original file name, content-type,
            // content-disposition, chunked "streaming" and cache-control
            // we're using low-level .serve() method
            const opts = {
                Bucket: s3Conf.bucket,
                Key: path
            };

            if (http.request.headers.range) {
                const vRef  = fileRef.versions[version];
                let range   = _.clone(http.request.headers.range);
                const array = range.split(/bytes=([0-9]*)-([0-9]*)/);
                const start = parseInt(array[1]);
                let end = parseInt(array[2]);
                if (isNaN(end)) {
                    // Request data from AWS:S3 by small chunks
                    end = (start + this.chunkSize) - 1;
                    if (end >= vRef.size) {
                        end = vRef.size - 1;
                    }
                }
                opts.Range = `bytes=${start}-${end}`;
                http.request.headers.range = `bytes=${start}-${end}`;
            }

            const fileColl = this;
            s3.getObject(opts, function(error) {
                if (error) {
                    console.error(error);
                    if (!http.response.finished) {
                        http.response.end();
                    }
                } else {
                    console.log(getObject);
                    if (http.request.headers.range && this.httpResponse.headers['content-range']) {
                        // Set proper range header in according to what is returned from AWS:S3
                        http.request.headers.range = this.httpResponse.headers['content-range'].split('/')[0].replace('bytes ', 'bytes=');
                    }

                    const dataStream = new stream.PassThrough();
                    fileColl.serve(http, fileRef, fileRef.versions[version], version, dataStream);
                    dataStream.end(this.data.Body);
                }
            });

            return true;
        }
        // While file is not yet uploaded to AWS:S3
        // It will be served file from FS
        return false;
    }
});

// Intercept FilesCollection's remove method to remove file from AWS:S3
const _origRemove = BizVideo.remove;
BizVideo.remove = function(search) {
    const cursor = this.collection.find(search);
    cursor.forEach((fileRef) => {
        _.each(fileRef.versions, (vRef) => {
            if (vRef && vRef.meta && vRef.meta.pipePath) {
                // Remove the object from AWS:S3 first, then we will call the original FilesCollection remove
                s3.deleteObject({
                    Bucket: s3Conf.bucket,
                    Key: vRef.meta.pipePath,
                }, (error) => {
                    bound(() => {
                        if (error) {
                            console.error(error);
                        }
                    });
                });
            }
        });
    });

    //remove original file from database
    _origRemove.call(this, search);
};

} else { throw new Meteor.Error(401, 'Missing Meteor file settings'); }

client/videoupload.js

import { Meteor } from 'meteor/meteor'; import { FilesCollection } from 'meteor/ostrio:files';

export const BizVideo = new FilesCollection({ collectionName: 'bussVideo', allowClientCode: false, chunkSize: 1024 * 1024 });

file where I have implemented it:

import { ReactiveVar } from 'meteor/reactive-var'; import { Bert } from 'meteor/themeteorchef:bert'; import { BizVideo } from '/imports/videoUploadClient/videoUpload.js';

var uploader = new ReactiveVar(); Template.vendorImagesVideos.onCreated(function() { this.currentUpload = new ReactiveVar(false); this.subscribe('getBizVideo'); });

Template.vendorImagesVideos.helpers({ currentUpload: function() { return Template.instance().currentUpload.get(); },

files: function() {
    var businessLink = FlowRouter.getParam('businessLink');
    var bussData = Business.findOne({"businessLink":businessLink});
    if(bussData){
        var data = BizVideo.find({"_id":bussData.businessVideo}).fetch();
        return data;
    }
},  

}

Template.vendorImagesVideos.events({ 'change #fileInput'(e, template) { if (e.currentTarget.files && e.currentTarget.files[0]) { var businessLink = FlowRouter.getParam('businessLink'); var bussData = Business.findOne({"businessLink":businessLink}); if(bussData.businessVideo){ Bert.alert('Only One can be upload','danger','growl-top-right'); }else{

          // We upload only one file, in case
          // multiple files were selected
          const upload = BizVideo.insert({
            file: e.currentTarget.files[0],
            streams: 'dynamic',
            chunkSize: 'dynamic'
          }, false);

          upload.on('start', function () {
            template.currentUpload.set(this);
          });

          upload.on('end', function (error, fileObj) {
            if (error) {
              alert('Error during upload: ' + error);
            } else {
              alert('File "' + fileObj._id + '" successfully uploaded');
                Meteor.call("updateVendorBulkVideo", businessLink,fileObj._id,
                  function(error, result) { 
                      if(error) {
                          console.log ('Error Message: ' +error ); 
                      }else{
                              // process.exit();
                      }
                });
            }
            template.currentUpload.set(false);
          });

          upload.start();
        }
    }
},

}

Server/main.js

import { BizVideo } from '/imports/videoUploadserver/videoUpload.js'; Meteor.publish('getBizVideo', function() { return BizVideo.find().cursor; });

I am able to see the video till the time it is in director. Please let me know where I am going wrong. Thanks in advance.