stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.62k stars 224 forks source link

Can't express ObjectId as a SQL literal for update ops #9

Closed addisonj closed 11 years ago

addisonj commented 11 years ago

As mentioned in #8, it looks like another case is not being handled in updates.

Stack here:

DEBUG MoSQL: processing op: {"ts"=>seconds: 1360870248, increment: 2, "h"=>-5242060077508462325, "v"=>2, "op"=>"u", "ns"=>"user.favorites", "o2"=>{"_id"=>BSON::ObjectId('511d3b6677f7ebd6d1c9c2e6')}, "o"=>{"$set"=>{"userId"=>BSON::ObjectId('511d3a8677f7ebd6d1c9c257')}}}
DEBUG MoSQL: resync user.favorites: 511d3b6677f7ebd6d1c9c2e6 (update was: {"$set"=>{"userId"=>BSON::ObjectId('511d3a8677f7ebd6d1c9c257')}})
/var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:1127:in `literal_other_append': can't express BSON::ObjectId('511d3b6677f7ebd6d1c9c2e6') as a SQL literal (Sequel::Error)
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:117:in `literal_append'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:465:in `complex_expression_sql_append'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/adapters/shared/postgres.rb:1057:in `complex_expression_sql_append'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/sql.rb:92:in `to_s_append'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:1087:in `literal_expression_append'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:95:in `literal_append'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:1319:in `select_where_sql'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:835:in `block in clause_sql'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:835:in `each'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:835:in `clause_sql'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/sql.rb:15:in `delete_sql'
        from /var/lib/gems/1.9.1/gems/sequel-3.44.0/lib/sequel/dataset/actions.rb:131:in `delete'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/lib/mosql/cli.rb:260:in `sync_object'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/lib/mosql/cli.rb:293:in `handle_op'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/lib/mosql/cli.rb:250:in `block in optail'
        from /var/lib/gems/1.9.1/gems/mongoriver-0.1.0/lib/mongoriver/abstract_persistent_tailer.rb:27:in `block in stream'
        from /var/lib/gems/1.9.1/gems/mongoriver-0.1.0/lib/mongoriver/tailer.rb:94:in `stream'
        from /var/lib/gems/1.9.1/gems/mongoriver-0.1.0/lib/mongoriver/abstract_persistent_tailer.rb:26:in `stream'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/lib/mosql/cli.rb:249:in `optail'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/lib/mosql/cli.rb:143:in `run'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/lib/mosql/cli.rb:16:in `run'
        from /var/lib/gems/1.9.1/gems/mosql-0.1.0/bin/mosql:7:in `<top (required)>'
        from /usr/local/bin/mosql:19:in `load'
        from /usr/local/bin/mosql:19:in `<main>'

Awesome project by the way :)

nelhage commented 11 years ago

Thanks for the report, and the compliments! I see the bug, working on a fix now.

nelhage commented 11 years ago

Does that fix it for you?

addisonj commented 11 years ago

Seems to be running very smoothly now!

Thanks!

Also, I need to be able to grab properties from nested objects. I am going to try and see if I can put something together and send off a PR in the next few days.

nelhage commented 11 years ago

Great, that's been a multiply-requested feature, and I'd love to merge it, I just haven't had the time yet.

christinang89 commented 10 years ago

Hey @nelhage , I'm facing the same issues as @addisonj above and this is the error I'm seeing:

INFO Mongoriver: Starting oplog stream from seconds: 1407538029, increment: 0
/usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1228:in `literal_other_append': can't express #<BSON::DBRef:0x007fe5cbc85060 @namespace="accounts", @object_id=BSON::ObjectId('53aa28bee4b008bb2ef7d7cd')> as a SQL literal (Sequel::Error)
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:107:in `literal_append'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1536:in `block in update_set_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1528:in `each'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:1528:in `update_set_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:228:in `_update_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/sql.rb:174:in `update_sql'
    from /usr/local/lib/ruby/gems/2.1.0/gems/sequel-4.12.0/lib/sequel/dataset/actions.rb:769:in `update'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/sql.rb:50:in `upsert!'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/sql.rb:38:in `upsert_ns'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:195:in `block in handle_op'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:39:in `unsafe_handle_exceptions'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:194:in `handle_op'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:154:in `block in optail'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mongoriver-0.3.1/lib/mongoriver/abstract_persistent_tailer.rb:28:in `block in stream'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mongoriver-0.3.1/lib/mongoriver/tailer.rb:107:in `stream'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mongoriver-0.3.1/lib/mongoriver/abstract_persistent_tailer.rb:27:in `stream'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/streamer.rb:153:in `optail'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/cli.rb:161:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/lib/mosql/cli.rb:16:in `run'
    from /usr/local/lib/ruby/gems/2.1.0/gems/mosql-0.3.1/bin/mosql:5:in `<top (required)>'
    from /usr/local/bin/mosql:23:in `load'
    from /usr/local/bin/mosql:23:in `<main>'