mongo-load-stream doesn't do anything

[kramer65]

I'm quite thrilled that I found mongo-dump-stream and I'm currently testing it out. From a remote server Ubuntu 14.04 I've successfully dumped our production mongodb into a file on a backup server (alsuo Ubuntu 14.04) using

ssh -C ubuntu@52.xx.xx.xxx mongo-dump-stream mongodb://127.0.0.1:27017/thedatabase | gzip -c > mongodb.gz

this created a file of 2.4 GB:

$ ls -lh
total 2.4G
-rw-rw-r-- 1 ubuntu ubuntu 2.4G Dec 24 15:46 mongodb.gz

I now want to load this stream into the mongodb of the backup server, which I try using:

gunzip -c mongodb.gz | mongo-load-stream mongodb://127.0.0.1:27017/thebackup

This command gives no error and ends within a second. But when I open the mongo shell I don't see the expected thebackup database:

$ mongo
MongoDB shell version: 2.4.9
connecting to: test
> show dbs;
local   0.078125GB

To make sure the unzipping goes well I tried gunziping the file first and if I pipe it through less (gunzip -c mongodb.gz | less) I see the following:

{ [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' }
R^@^@^@^Btype^@^R^@^@^@mongo-dump-stream^@^Bversion^@^B^@^@^@2^@^BendOfCollection^@^M^@^@^@C42uzh+cSiI=^@^@<95>^@^@^@^Btype^@^K^@^@^@collection^@^Bname^@^N^@^@^@user_document^@^Dindexes^@Z^@^@^@^C0^@R^@^@^@^Pv^@^A^@^@^@^Ckey^@^N^@^@^@^P_id^@^A^@^@^@^@^Bns^@^\^@^@^@ticketcatcher.user_document^@^Bname^@^E^@^@^@_id_^@^@^@^@<A1>^E^@^@^G_id^@V^ER-<F7>!<F5>^H!<80>        N       created^@ ^@<C9>^DP^A^@^@^Bfilename^@^A^@^@^@^@^Ebinary^@4^E^@^@^@<FF><D8><FF><E0>^@^PJFIF^@^A^B^@^@^A^

I don't know if the error on the first line is a problem, but it seems to extract perfectly fine into a file of 2.9GB

I also tried piping the extracted mongodb file to mongo-load-streamwith cat mongodb | mongo-load-stream mongodb://127.0.0.1:27017/thebackup, but with the same result; it finishes immediately and nothing seems to be imported in mongodb.

Does anybody know why mongo-load-stream doesn't seem to do anything for me? All tips are welcome!

[boutell]

First line is definitely a problem. It's an error message and not part of the data. Not sure why node wrote it to stdout rather than stderr which pretty much guarantees wrecked output. But it's not an error in mongo-dump-stream itself. On Dec 28, 2015 7:41 AM, "kramer65" notifications@github.com wrote:

I'm quite thrilled that I found mongo-dump-stream and I'm currently testing it out From a remote server Ubuntu 1404 I've successfully dumped our production mongodb into a file on a backup server (alsuo Ubuntu 1404) using

ssh -C ubuntu@52xxxxxxx mongo-dump-stream mongodb://127001:27017/thedatabase | gzip -c > mongodbgz

this created a file of 24 GB:

$ ls -lh total 24G -rw-rw-r-- 1 ubuntu ubuntu 24G Dec 24 15:46 mongodbgz

I now want to load this stream into the mongodb of the backup server, which I try using:

gunzip -c mongodbgz | mongo-load-stream mongodb://127001:27017/thebackup

This command gives no error and ends within a second But when I open the mongo shell I don't see the expected thebackup database:

$ mongo MongoDB shell version: 249 connecting to: test

show dbs; local 0078125GB

To make sure the unzipping goes well I tried gunziping the file first and if I pipe it through less (gunzip -c mongodbgz | less) I see the following:

{ [Error: Cannot find module '/build/Release/bson'] code: 'MODULE_NOT_FOUND' } R^@^@^@^Btype^@^R^@^@^@mongo-dump-stream^@^Bversion^@^B^@^@^@2^@^BendOfCollection^@^M^@^@^@C42uzh+cSiI=^@^@95>^@^@^@^Btype^@^K^@^@^@collection^@^Bname^@^N^@^@^@user_document^@^Dindexes^@Z^@^@^@^C0^@R^@^@^@^Pv^@^A^@^@^@^Ckey^@^N^@^@^@^P_id^@^A^@^@^@^@^Bns^@^^@^@^@ticketcatcheruser_document^@^Bname^@^E^@^@^@_id_^@^@^@^@<A1^E^@^@^G_id^@V^ER-!^H!<80> N created^@ ^@C9>^DP^A^@^@^Bfilename^@^A^@^@^@^@^Ebinary^@4^E^@^@^@<FF^@^PJFIF^@^A^B^@^@^A^

I don't know if the error on the first line is a problem, but it seems to extract perfectly fine into a file of 29GB

I also tried piping the extracted mongodb file to mongo-load-streamwith cat mongodb | mongo-load-stream mongodb://127001:27017/thebackup, but with the same result; it finishes immediately and nothing seems to be imported in mongodb

Does anybody know why mongo-load-stream doesn't seem to do anything for me? All tips are welcome!

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/11.

[boutell]

That said I will keep this open to look at why there isn't better error reporting when the input stream is bad.

Note that mongodb 3.2 has some built in piping capabilities in its utilities. On Dec 28, 2015 7:57 AM, "Tom Boutell" tom@punkave.com wrote:

First line is definitely a problem. It's an error message and not part of the data. Not sure why node wrote it to stdout rather than stderr which pretty much guarantees wrecked output. But it's not an error in mongo-dump-stream itself. On Dec 28, 2015 7:41 AM, "kramer65" notifications@github.com wrote:

I'm quite thrilled that I found mongo-dump-stream and I'm currently testing it out From a remote server Ubuntu 1404 I've successfully dumped our production mongodb into a file on a backup server (alsuo Ubuntu 1404) using

ssh -C ubuntu@52xxxxxxx mongo-dump-stream mongodb://127001:27017/thedatabase | gzip -c > mongodbgz

this created a file of 24 GB:

$ ls -lh total 24G -rw-rw-r-- 1 ubuntu ubuntu 24G Dec 24 15:46 mongodbgz

I now want to load this stream into the mongodb of the backup server, which I try using:

gunzip -c mongodbgz | mongo-load-stream mongodb://127001:27017/thebackup

This command gives no error and ends within a second But when I open the mongo shell I don't see the expected thebackup database:

$ mongo MongoDB shell version: 249 connecting to: test

show dbs; local 0078125GB

To make sure the unzipping goes well I tried gunziping the file first and if I pipe it through less (gunzip -c mongodbgz | less) I see the following:

{ [Error: Cannot find module '/build/Release/bson'] code: 'MODULE_NOT_FOUND' } R^@^@^@^Btype^@^R^@^@^@mongo-dump-stream^@^Bversion^@^B^@^@^@2^@^BendOfCollection^@^M^@^@^@C42uzh+cSiI=^@^@95>^@^@^@^Btype^@^K^@^@^@collection^@^Bname^@^N^@^@^@user_document^@^Dindexes^@Z^@^@^@^C0^@R^@^@^@^Pv^@^A^@^@^@^Ckey^@^N^@^@^@^P_id^@^A^@^@^@^@^Bns^@^^@^@^@ticketcatcheruser_document^@^Bname^@^E^@^@^@_id_^@^@^@^@<A1^E^@^@^G_id^@V^ER-!^H!<80> N created^@ ^@C9>^DP^A^@^@^Bfilename^@^A^@^@^@^@^Ebinary^@4^E^@^@^@<FF^@^PJFIF^@^A^B^@^@^A^

I don't know if the error on the first line is a problem, but it seems to extract perfectly fine into a file of 29GB

I also tried piping the extracted mongodb file to mongo-load-streamwith cat mongodb | mongo-load-stream mongodb://127001:27017/thebackup, but with the same result; it finishes immediately and nothing seems to be imported in mongodb

Does anybody know why mongo-load-stream doesn't seem to do anything for me? All tips are welcome!

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/11.

[kramer65]

Thanks for your reply. I now indeed see that the error is simply the output of

mongo-dump-stream mongodb://127.0.0.1:27017/thedatabase | less

I searched around and found some suggested solutions (such as in this stackoverflow question), but I can't figure out how to solve it. Any idea how I can solve this Cannot find module error?

Also thanks for the tip on mongodb 3.2, using the basic mongodump sounds good. Version 3.2 is a bit too new to my taste though so until I'm switching my servers over to Ubuntu 16.04 in a year or so, I'll stick with the currently installed version (now v2.4.9). So until that time I would still love to use mongo-dump-stream. So any help would still be appreciated! :-)

[boutell]

Well, the right place to start would be to figure out what outputs that error and open an issue asking why it's written to stdout instead of stderr.

On Mon, Dec 28, 2015 at 11:52 AM, kramer65 notifications@github.com wrote:

Thanks for your reply. I now indeed see that the error is simply the output of

mongo-dump-stream mongodb://127.0.0.1:27017/thedatabase | less

I searched around and found some suggested solutions (such as in this stackoverflow question http://stackoverflow.com/questions/28651028/cannot-find-module-build-release-bson-code-module-not-found-js-bson), but I can't figure out how to solve it. Any idea how I can solve this Cannot find module error?

Also thanks for the tip on mongodb 3.2, using the basic mongodump sounds good. Version 3.2 is a bit too new to my taste though so until I'm switching my servers over to Ubuntu 16.04 in a year or so, I'll stick with the currently installed version (now v2.4.9). So until that time I would still love to use mongo-dump-stream. So any help would still be appreciated! :-)

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/11#issuecomment-167602433 .

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

[boutell]

(Probably on node itself, but I'm really not sure.)

On Mon, Dec 28, 2015 at 12:49 PM, Tom Boutell tom@punkave.com wrote:

Well, the right place to start would be to figure out what outputs that error and open an issue asking why it's written to stdout instead of stderr.

On Mon, Dec 28, 2015 at 11:52 AM, kramer65 notifications@github.com wrote:

Thanks for your reply. I now indeed see that the error is simply the output of

mongo-dump-stream mongodb://127.0.0.1:27017/thedatabase | less

I searched around and found some suggested solutions (such as in this stackoverflow question http://stackoverflow.com/questions/28651028/cannot-find-module-build-release-bson-code-module-not-found-js-bson), but I can't figure out how to solve it. Any idea how I can solve this Cannot find module error?

Also thanks for the tip on mongodb 3.2, using the basic mongodump sounds good. Version 3.2 is a bit too new to my taste though so until I'm switching my servers over to Ubuntu 16.04 in a year or so, I'll stick with the currently installed version (now v2.4.9). So until that time I would still love to use mongo-dump-stream. So any help would still be appreciated! :-)

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/11#issuecomment-167602433 .

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

[xdc0]

Also noticing this on mongo 3.2

When I pipe mongo-dump-stream to less, the very first line reads: js-bson: Failed to load c++ bson extension, using pure JS version

[boutell]

This is not a bug in mongo-dump-stream. You have noise on stdout because you don't have a proper compiler in your environment and the mongo driver prints a warning when it's built as the pure JS version. This also makes it slower, so I recommend addressing that.

On Thu, Apr 28, 2016 at 4:17 PM, Chuy Martinez notifications@github.com wrote:

Also noticing this on mongo 3.2

When I pipe mongo-dump-stream to less, the very first line reads: js-bson: Failed to load c++ bson extension, using pure JS version

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/11#issuecomment-215549339

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

[boutell]

To be clear, the Rx is:

• Install your compiler
• npm rebuild
• Profit!