search

stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.63k stars 225 forks source link

Doesn't handle BSON::DBRef correctly #50

Closed nelhage closed 10 years ago

nelhage commented 10 years ago

See https://github.com/stripe/mosql/issues/9#issuecomment-51666800

@christinang89 -- looks like the problem is that your data has a BSON::ObjectRef, which we haven't implemented support for. I'm not sure what the right thing to do with those in general is, but I suspect that the right answer is to just extract out the referenced _id and store that in the same form as the normal _id fields, so that you can do JOINs on them in SQL. Would that work for your use case?

christinang89 commented 10 years ago

@nelhage thanks!

Yes, that would work for my use case. However, what would you suggest as a simple way to extract out the referenced _id?

Also, it seems that this is causing the op-tail not to work and I have to manually reimport all my data on a daily basis. (For now, I'm using crontab to do this twice a day as a stop gap solution).

Do you have a better/ easier/ more elegant solution? Thanks for your help!

nelhage commented 10 years ago

@christinang89 Are you able to test the commit I just pushed in your environment? If that works, I'll merge and mark this as closed.

christinang89 commented 10 years ago

Hmm, unless I'm doing something wrong, it's still throwing this error:

 INFO Mongoriver: Saved timestamp: seconds: 1407784998, increment: 1 (2014-08-11 12:23:18 -0700)
/usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1228:in `literal_other_append': can't express #<BSON::DBRef:0x007f9b4b51c3e8 @namespace="accounts", @object_id=BSON::ObjectId('53aa28bee4b008bb2ef7d7cd')> as a SQL literal (Sequel::Error)
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:107:in `literal_append'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1536:in `block in update_set_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1528:in `each'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1528:in `update_set_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:228:in `_update_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:174:in `update_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/actions.rb:769:in `update'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/sql.rb:51:in `upsert!'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/sql.rb:39:in `upsert_ns'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:213:in `block in handle_op'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:39:in `unsafe_handle_exceptions'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:212:in `handle_op'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:172:in `block in optail'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mongoriver-0.3.1/lib/mongoriver/abstract_persistent_tailer.rb:28:in `block in stream'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mongoriver-0.3.1/lib/mongoriver/tailer.rb:107:in `stream'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mongoriver-0.3.1/lib/mongoriver/abstract_persistent_tailer.rb:27:in `stream'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:171:in `optail'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/cli.rb:165:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/cli.rb:16:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/bin/mosql:5:in `<top (required)>'
    from /usr/local/bin/mosql:23:in `load'
    from /usr/local/bin/mosql:23:in `<main>'
nelhage commented 10 years ago

Hm, your backtrace suggests you're still running the installed 0.3.1 version. Did you clone the branch and try running directly out of there? I'm pretty sure this should fix it, so I'm happy to merge and cut a gem if that's easier for you.

christinang89 commented 10 years ago

Yea I tried to run directly out of there, but it is likely I did it wrong. (Sorry I'm a beginner!)

Yea maybe if you just go ahead and merge it, I can try and report issues, if any! :)

Thank you!! :D

christinang89 commented 10 years ago

Hey @nelhage so i managed to clone it correctly, just tested it and verified that it works smoothly! :)

nelhage commented 10 years ago

awesome! Sorry, I was going to cut a release but got pulled away. I'll merge and do that now.