search

stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.62k stars 224 forks source link

`block in bulk_upsert': undefined method `primary_sql_key_for_ns' for #<MoSQL::CLI:0x00000000e8e428> (NoMethodError) #17

Closed mrgordon closed 11 years ago

mrgordon commented 11 years ago 
 INFO MoSQL: Imported 29397000 rows (8254.963520826s, 4295.059250159007s SQL)...
 INFO MoSQL: Imported 29398000 rows (8255.24602006s, 4295.177102176007s SQL)...
somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:167:in `block in bulk_upsert': undefined method `primary_sql_key_for_ns' for #<MoSQL::CLI:0x00000000e8e428> (NoMethodError)
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:164:in `each'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:164:in `rescue in bulk_upsert'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:159:in `bulk_upsert'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:233:in `block (4 levels) in import_collection'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:190:in `track_time'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:232:in `block (3 levels) in import_collection'
        from /usr/local/rvm/gems/ruby-1.9.3-p125/gems/mongo-1.8.3/lib/mongo/cursor.rb:271:in `each'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:227:in `block (2 levels) in import_collection'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:175:in `block in with_retries'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:173:in `times'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:173:in `with_retries'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:226:in `block in import_collection'
        from /usr/local/rvm/gems/ruby-1.9.3-p125/gems/mongo-1.8.3/lib/mongo/collection.rb:269:in `find'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:225:in `import_collection'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:208:in `block (2 levels) in initial_import'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:206:in `each'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:206:in `block in initial_import'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:202:in `each'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:202:in `initial_import'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:143:in `run'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/lib/mosql/cli.rb:16:in `run'
        from somedir/.bundler/ruby/1.9.1/mosql-5945b627f05c/bin/mosql:5:in `<top (required)>'
        from /usr/local/rvm/gems/ruby-1.9.3-p125/bin/mosql:23:in `load'
        from /usr/local/rvm/gems/ruby-1.9.3-p125/bin/mosql:23:in `<main>'
worker_ui@app6:/home/mgordon/mosql$ bundle exec mosql --sql postgres://postgres@myhost/mydb --mongo mongodb://mongohost:27017?slaveOk=true
 INFO Mongoriver: Starting oplog stream from seconds: 1363616697, increment: 0
 INFO Mongoriver: Saved timestamp: seconds: 1363616867, increment: 70 (2013-03-18 14:27:47 +0000)
 INFO Mongoriver: Saved timestamp: seconds: 1363616928, increment: 1 (2013-03-18 14:28:48 +0000)
 INFO Mongoriver: Saved timestamp: seconds: 1363616989, increment: 1 (2013-03-18 14:29:49 +0000)
 INFO Mongoriver: Saved timestamp: seconds: 1363617050, increment: 1 (2013-03-18 14:30:50 +0000)

Saw this while trying to reimport data.

(Side question) Would you expect it to be safe to restart MoSQL if it stops running for a few days or does this generally require a reimport? Is its ability to catch up constrained by whether that many days of oplogs are available?

nelhage commented 11 years ago

Thanks for the report. I just pushed a fix to that crash.

If it stop running for a few days, whether or not resuming tailing is safe does depend on how much oplog is available. You can tell this via db.getReplicationInfo() in the mongo shell, or by looking at the local.oplog.rs collection directly.

Of course, even if you can catch up, if you're far enough behind and you have enough write volume, it might be faster to just reimport regardless.

mrgordon commented 11 years ago

Thanks for the fix and the info!

On Mar 25, 2013, at 7:20 PM, Nelson Elhage notifications@github.com wrote:

Thanks for the report. I just pushed a fix to that crash.

If it stop running for a few days, whether or not resuming tailing is safe does depend on how much oplog is available. You can tell this via db.getReplicationInfo() in the mongo shell, or by looking at the local.oplog.rs collection directly.

Of course, even if you can catch up, if you're far enough behind and you have enough write volume, it might be faster to just reimport regardless.

— Reply to this email directly or view it on GitHubhttps://github.com/stripe/mosql/issues/17#issuecomment-15437384 .