aheckmann / gridfs-stream

Easily stream files to and from MongoDB
MIT License
615 stars 120 forks source link

How can we avoid storing duplicates in mongo database using grid fs #145

Closed jelordreygulle closed 5 years ago

jelordreygulle commented 5 years ago

How can we avoid storing duplicates in mongo database using grid fs ?

In my code below I directly pipe the request stream to gfs write stream but I cannot check if the file exists or not so duplicate files are being stored in the database . Any idea using this gridfs stream ?


 var download = function (url, dest, filename, callback) {

                    const file = filename
                    const fileStorage = gfs.createWriteStream({ filename: file });

                    request.get(url)
                        .on('error', function (err) { console.log(err) })
                        .pipe(fileStorage)
                        .on('close', callback);

                };

                List.forEach(function (str) {
                    var filename = str.split('/').pop();
                    console.log('Downloading ' + filename);
                    promises.push(new Promise(function p(resolve, reject) {
                        download(str, filename, function () {
                            console.log('Finished Downloading' + "" + filename);
                            resolve(filename);
                        }); // add error callback also, call reject() on error
                    }
                    ));
                });