Support for GridFS (mongodb)

[aperezl]

Hi,

I have started working on a backend storage for gridfs. At the moment I have done some tests and all have been satisfactory.

My idea is to have a new storage for apostrophecms, although at the moment we also have to make changes in the apostrophe-attachments module. (I wish this was not necessary).

My question is, do you think it might be interesting to have this new backend storage? Is there a possibility that it will be published?

Thanks.

[boutell]

Sure, why not? As long as it is a correct implementation that would be fine. Mongo hosting is usually not priced in a way that makes it attractive for large file storage, but on a site with a modest number of files it could be a way to avoid adding s3 to the picture.

The biggest downside, besides price, is that the files would then have to be streamed through mongo and node whenever anyone looks at them, whereas s3, azure and local (via nginx) can all be served without all that overhead. But, you could put a CDN in front of the site, configured to cache anything in /uploads/attachments.

What changes do you need in the attachments module?

On Oct 11, 2018 10:05 AM, "Antonio Manuel Pérez López" < notifications@github.com> wrote:

Hi,

I have started working on a backend storage for gridfs. At the moment I have done some tests and all have been satisfactory.

My idea is to have a new storage for apostrophecms, although at the moment we also have to make changes in the apostrophe-attachments module. (I wish this was not necessary).

My question is, do you think it might be interesting to have this new backend storage? Is there a possibility that it will be published?

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/punkave/uploadfs/issues/45, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9faI01rMOqCrDgIFVa95K-PnGZJS7ks5uj3pvgaJpZM4XX-th .

[aperezl]

Hi,

I need to inject gridfSBucket and express from attachments to be used by uploadfs (I do not want to create new dependencies in this module). It may not be necessary at the end and only 2 files would be needed in uploadfs (the backend and the test)

At the moment I only have the basics, but I have been able to create and visualize files in Heroku with mLab.

I agree that mongo is not an optimal solution for a large CMS, but for a small blog or academic use it may be fine.

Thanks for the opportunity

[boutell]

Right... Mongo is a great database solution for a CMS of any size, but for uploaded media file storage it probably makes sense only for smaller sites.

As for injecting the mongodb driver, I see your point. You could make a separate connection in uploadfs, but to save resources it makes sense to accept an existing db object, and that raises the question of how apostrophe-attachments knows to provide it to you. We'd accept a PR to include db in the uploadfs options if storage is set to gridfs; that would be a simple and practical fix.

You also need Express, yes, so you can set up a way for the files to actually be served. Once again, it would be fine for apostrophe-attachments to push self.apos.app to uploadfs if storage is set to gridfs.

[aperezl]

That is exactly my approach :) (A little example)

    if(self.uploadfsSettings.backend === 'gridfs') {
      var GridFSBucket =  require('mongodb').GridFSBucket
      var bucket = new GridFSBucket(self.apos.db, { bucketName: 'tracks' })
      _.merge(self.uploadfsSettings, {
        bucket: bucket,
        app: self.apos.app
      });
    }

[boutell]

I would not set bucketName by default, because it has a built-in default in mongodb (fs). I would suggest adding that property only if options.uploadfs.bucketName is set. The general idea makes sense to me.

On Tue, Oct 16, 2018 at 1:57 PM Antonio Manuel Pérez López < notifications@github.com> wrote:

That is exactly my approach :) (A little example)

if(self.uploadfsSettings.backend === 'gridfs') {
  var GridFSBucket =  require('mongodb').GridFSBucket
  var bucket = new GridFSBucket(self.apos.db, { bucketName: 'tracks' })
  _.merge(self.uploadfsSettings, {
    bucket: bucket,
    app: self.apos.app
  });
}

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/punkave/uploadfs/issues/45#issuecomment-430334925, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9fXWh6Bm_-dktyTln1ywJkhLYxAcgks5ulh3ugaJpZM4XX-th .

--

Thomas Boutell, Chief Software Architect P'unk Avenue | (215) 755-1330 | punkave.com

[boutell]

To be accepted as a PR to uploadfs, the solution must handle the enable and disable methods successfully. That could be as simple as a naming convention that your serving route pays attention to when deciding whether to serve something.

On Tue, Oct 16, 2018 at 2:28 PM Tom Boutell tom@punkave.com wrote:

I would not set bucketName by default, because it has a built-in default in mongodb (fs). I would suggest adding that property only if options.uploadfs.bucketName is set. The general idea makes sense to me.

On Tue, Oct 16, 2018 at 1:57 PM Antonio Manuel Pérez López < notifications@github.com> wrote:

That is exactly my approach :) (A little example)

if(self.uploadfsSettings.backend === 'gridfs') {
  var GridFSBucket =  require('mongodb').GridFSBucket
  var bucket = new GridFSBucket(self.apos.db, { bucketName: 'tracks' })
  _.merge(self.uploadfsSettings, {
    bucket: bucket,
    app: self.apos.app
  });
}

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/punkave/uploadfs/issues/45#issuecomment-430334925, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9fXWh6Bm_-dktyTln1ywJkhLYxAcgks5ulh3ugaJpZM4XX-th .

--

Thomas Boutell, Chief Software Architect P'unk Avenue | (215) 755-1330 | punkave.com

--

Thomas Boutell, Chief Software Architect P'unk Avenue | (215) 755-1330 | punkave.com