search

punkave / mongo-dump-stream

Pipe entire mongodb databases through the shell, or to a node stream. It's what you wanted mongodump to be.
MIT License
51 stars 11 forks source link

v0.1.1 tests failing on OSX & Linux #1

Closed kylestetz closed 9 years ago

kylestetz commented 9 years ago

Just wanted to note that it's not travis's fault the tests are failing— I am seeing the same results on my machine with mongo v2.6.5.

Here's the output of the travis test, and here's my (identical) output:

mongo-dump-stream
    ✓ populate the test database (607ms)
    ✓ can dump the test database to a file (139ms)
    1) can load the file into a second database
    2) the contents of the second database are correct

  2 passing (3s)
  2 failing

  1) mongo-dump-stream can load the file into a second database:
     Error: timeout of 2000ms exceeded
      at null.<anonymous> (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mocha/lib/runnable.js:159:19)
      at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)

  2) mongo-dump-stream the contents of the second database are correct:
     Uncaught AssertionError: false == true
      at /Users/kylestetz/Sites/mongo-dump-stream/tests/test.js:111:13
      at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/cursor.js:197:9
      at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/cursor.js:229:48
      at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/cursor.js:905:28
      at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/db.js:1903:9
      at Server.Base._callHandler (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/base.js:453:41)
      at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/server.js:487:18
      at MongoReply.parseBody (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
      at null.<anonymous> (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/server.js:445:20)
      at emit (events.js:95:17)
      at null.<anonymous> (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:207:13)
      at emit (events.js:98:17)
      at Socket.<anonymous> (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/connection.js:357:18)
      at Socket.emit (events.js:95:17)
      at Socket.<anonymous> (_stream_readable.js:764:14)
      at Socket.emit (events.js:92:17)
      at emitReadable_ (_stream_readable.js:426:10)
      at emitReadable (_stream_readable.js:422:5)
      at readableAddChunk (_stream_readable.js:165:9)
      at Socket.Readable.push (_stream_readable.js:127:10)
      at TCP.onread (net.js:528:21)
boutell commented 9 years ago

Ruh-roh. This could be a node 0.10 thing (I wrote this with 0.12 and I semi-trusted streams to sort of work for once), but it could just be a time-for-the-test thing, so let me try it on an 0.10 server, and if code changes are needed, make them there.

On Sun, Mar 1, 2015 at 7:49 AM, Kyle Stetz notifications@github.com wrote:

Just wanted to note that it's not travis's fault the tests are failing— I am seeing the same results on my machine with mongo v2.6.5.

Here's the output of the travis test https://travis-ci.org/punkave/mongo-dump-stream/jobs/52619888, and here's my (identical) output:

mongo-dump-stream ✓ populate the test database (607ms) ✓ can dump the test database to a file (139ms) 1) can load the file into a second database 2) the contents of the second database are correct

2 passing (3s) 2 failing

1) mongo-dump-stream can load the file into a second database: Error: timeout of 2000ms exceeded at null. (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mocha/lib/runnable.js:159:19) at Timer.listOnTimeout as ontimeout

2) mongo-dump-stream the contents of the second database are correct: Uncaught AssertionError: false == true at /Users/kylestetz/Sites/mongo-dump-stream/tests/test.js:111:13 at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/cursor.js:197:9 at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/cursor.js:229:48 at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/cursor.js:905:28 at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/db.js:1903:9 at Server.Base._callHandler (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/base.js:453:41) at /Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/server.js:487:18 at MongoReply.parseBody (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5) at null. (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/server.js:445:20) at emit (events.js:95:17) at null. (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:207:13) at emit (events.js:98:17) at Socket. (/Users/kylestetz/Sites/mongo-dump-stream/node_modules/mongodb/lib/mongodb/connection/connection.js:357:18) at Socket.emit (events.js:95:17) at Socket. (_streamreadable.js:764:14) at Socket.emit (events.js:92:17) at emitReadable (_stream_readable.js:426:10) at emitReadable (_stream_readable.js:422:5) at readableAddChunk (_stream_readable.js:165:9) at Socket.Readable.push (_stream_readable.js:127:10) at TCP.onread (net.js:528:21)

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/1.

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

kylestetz commented 9 years ago

The tests are failing for 0.10, 0.11, and 0.12 (same failure) according to Travis. io.js fails as well.

boutell commented 9 years ago

It was just the time the tests were taking. They happened to run barely fast enough on my mac.

I dialed the size of the test database down to 100 documents per collection and now tests pass on a VPS.

Curious to see if it's fast enough to satisfy travis. We could dial up how long mocha waits as well.

On Sun, Mar 1, 2015 at 8:07 AM, Kyle Stetz notifications@github.com wrote:

The tests are failing for 0.10, 0.11, and 0.12 (same failure) according to Travis. io.js fails as well.

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/1#issuecomment-76596006 .

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

boutell commented 9 years ago

(Obviously, performance matters, so I should be thinking about ways to optimize that.)

On Sun, Mar 1, 2015 at 8:09 AM, Tom Boutell tom@punkave.com wrote:

It was just the time the tests were taking. They happened to run barely fast enough on my mac.

I dialed the size of the test database down to 100 documents per collection and now tests pass on a VPS.

Curious to see if it's fast enough to satisfy travis. We could dial up how long mocha waits as well.

On Sun, Mar 1, 2015 at 8:07 AM, Kyle Stetz notifications@github.com wrote:

The tests are failing for 0.10, 0.11, and 0.12 (same failure) according to Travis. io.js fails as well.

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/1#issuecomment-76596006 .

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

kylestetz commented 9 years ago

Passed in 0.12 and io.js, failed in 0.11 and 0.10 with the same error. Here's the output by Node version.

boutell commented 9 years ago

I suspect it's still marginal due to the time it takes. I will adjust the timeouts.

On Sun, Mar 1, 2015 at 8:11 AM, Kyle Stetz notifications@github.com wrote:

Passed in 0.12 and io.js, failed in 0.11 and 0.10 with the same error. Here's the output by Node version. https://travis-ci.org/punkave/mongo-dump-stream.

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/1#issuecomment-76596141 .

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

boutell commented 9 years ago

All green, phew.

I'm going to need to look at ways to better exploit async in this, take advantage of bulk mongo inserts etc.

On Sun, Mar 1, 2015 at 8:12 AM, Tom Boutell tom@punkave.com wrote:

I suspect it's still marginal due to the time it takes. I will adjust the timeouts.

On Sun, Mar 1, 2015 at 8:11 AM, Kyle Stetz notifications@github.com wrote:

Passed in 0.12 and io.js, failed in 0.11 and 0.10 with the same error. Here's the output by Node version. https://travis-ci.org/punkave/mongo-dump-stream.

— Reply to this email directly or view it on GitHub https://github.com/punkave/mongo-dump-stream/issues/1#issuecomment-76596141 .

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com

THOMAS BOUTELL, DEV & OPS P'UNK AVENUE | (215) 755-1330 | punkave.com