How to save to different databases based on userId

[kjami]

Hi,

Thanks for the simple yet wonderful package.

I don't have issue with the package but I have a question: I am looking to customise the package a little bit to suite our current application, so I have downloaded the package source code and added it as local package. The application's requirement is to store files into different databases based on company, which is determined by Meteor.userId().

For Mongo.Collection, one of my colleagues have implemented an extension method (Mongo.Collection.prototype.getUserCollection) to generate company collection if it is not already available. So we use myMongoCollection.getUserCollection(Meteor.userId) to get the company collection before calling the insert, find and other methods. But for FileCollection, it is very difficult for me to apply the same concept because, the insert method of FileCollection calls the base class's (Mongo.Collection's) insert method internally by using the Object.call(this, arguments). Can you please guide me in the right direction to solve my issue (where to start, what are things to consider etc.,).

Thanks in advance.

Regards, Kishor

[vsivsi]

Hi, do you really mean a different MongoDB database, or just a different collection within the same database? If it is truly different databases, then this is probably nearly impossible, because each database will require its own connection (this is an attribute of the node.js MongoDB driver).

If it is different collections within a database, I think this will still be difficult, but maybe doable based on what features of file-collection you need to use. You are correct in your instinct that it will require extensive changes to the file-collection source code, because much of the source depends on the coffeescript @ (equivalent of JS this) to refer to the current collection object. By your requirement, the way that the pub/sub and/or HTTP interfaces are managed and implemented will need to be changed in major ways.

I understand that these are your requirements, but I'm afraid that I'm not interested in supporting this use case for file-collection. Implementing this would make the code much more complex and difficult to understand and maintain, violating my goal to keep this package simple.

As a suggestion, you might instead look at producing an alternative gridFS storage adapter for the popular CollectionFS system. In that case, you can just write a new storage backend that meets your requirements without modifying all of the frontend code (e.g. the HTTP interface code). CollectionFS is much more complicated than file-collection, but one of the benefits of that complexity is that it is much more modular, making implementing a requirement like yours a potentially better match to its architecture.

[kjami]

Hi Vaughn,

Thanks for quick response. Yes, I mean a different database. For the default meteor collections, we are doing

Mongo.Collection.prototype.getUserCollection = function (userId) {
    this._companyCollections = this._companyCollections || { };

    if (Meteor.isClient) {
        return this;
    }

    if (!userId) {
        throw new Meteor.Error(403, "Cannot get the collection");
    } else {
        var company = null;

        //..
        //get company using the userId..

        if (!(company._id in this._companyCollections)) {
            var driver = new MongoInternals.RemoteCollectionDriver(company.mongoUrl); 

                   //since meteor collections cannot have same name, append the company id
            this._companyCollections[company._id] = new Mongo.Collection( this._name + '_' + company._id, { _driver: driver });

                    //change the underlying mongo collection's name to the original name
            this._companyCollections[company._id]._collection = driver.open(this._name, this._companyCollections[company._id]._connection);
            if (this.simpleSchema()) this._companyCollections[company._id].attachSchema(this.simpleSchema());
            //..
            //similarly attaching indexes and other information needed.
        }

        return this._companyCollections[company._id];
    };
};

//Declaration doesn't change..
Test = new Mongo.Collection('Test');

//But for CRUD operations, we do something like
Test.getUserCollection(Meteor.userId()).find();

In this way, we were successfully able to store data into different databases. You are right that this is bit hacky and not easily maintainable but since the industry in which we were working requires us to store data in different databases for different customers, we were left with no other option. So I want to know whether something like this is possible for me to do in your file-collection package as well. I don't want you to change your package, as I know it is not a standard way. It would be helpful if you give some guidance on how to proceed from here to make similar changes to FileCollection, so that I can do it locally in my application.

Regarding your other suggestion to try collectionFS, We have tried CollectionFS already and created a method similar to the one above (FS.Collection.prototype.getUserCollection). We were successful in saving the files to different databases. collection.files and collection.chunks are in different databases but collection.filerecords is still in the default meteor database. We are pretty satisfied with result. But we have encountered https://github.com/CollectionFS/Meteor-CollectionFS/issues/731 issue, when we tried to store a file from buffer on server side (This is required since we upload some files to server, convert them on server and store it back to collection). The issue is sporadic and is causing the application to crash and restart (I can say that this issue is only because of CollectionFS and not because of buffer or any other logic because, we have the same logic working without any issues when storing to s3 using slingshot). So we looked at alternatives because to debug the problem from huge source code of collectionFS will be very time consuming. We have chosen your package it is second most popular package after collectionFS with far less code and still actively developed. Hopefully, there is a way to get beyond this issue.

Any sort of guidance/help is very much appreciated. Thanks in advance,

[vsivsi]

I see.

Well, you are going to run into two issues that I can immediately think of. The first you already encountered. file-collection is of course written in Coffeescript. The Object.call(this, arguments) code you have encountered for fc.insert() is generated by the Coffeescript super operator. If you follow that link and look at the generated javascript on the right, you will see where that comes from. So the only way to eliminate it would be to re-implement file-collection to not be a sub-class of Mongo.collection. What a pain.

The second issue might be even bigger. Meteor uses an older version of the node.js MongoDB native driver. File-collection requires the newer 2.x driver because it implements node.js streams for data going in and out of gridFS. So, file-collection actually uses two drivers (and connections) to the database already. The "default" Meteor driver for basic operations (via the super operator calling methods on the parent Mongo.collection class), and the "newer" driver for operations that actually touch gridFS data.

Each file-collection instance creates and opens its own private connection to MongoDB: https://github.com/vsivsi/meteor-file-collection/blob/master/src/gridFS_server.coffee#L35 Which is then used to initialize the gridFS streaming interface: https://github.com/vsivsi/meteor-file-collection/blob/master/src/gridFS_server.coffee#L48

So in addition to redirecting the normal Meteor MongoDB methods to your own version using the 1.3.x native driver, you will also need to somehow need to do the same using the 2.x MongoDB driver for the streaming operations against the gridstore (Anything that touches @gfs, basically).

Good luck, this will probably be a major undertaking!

[kjami]

Yes Vaughn, looks like it is a major change. Thanks for your valuable time and input. I will see what I can do.

Have a good day ahead.