stripe-archive / mongoriver

A library for writing MongoDB oplog tailers.
MIT License
153 stars 33 forks source link

Support for Mongo driver >= 2.0 #15

Open karthikmuralidharan opened 8 years ago

karthikmuralidharan commented 8 years ago

Please correct me if I am wrong, but does the current gem support Mongo ruby driver v2 and above ? I had to downgrade it to 1.9.2 for it to work.

poislagarde commented 7 years ago

I know this is an old issue, but I was wondering how much work would be needed to update to mongo driver >= 2.0 ? I know there's some breaking changes, and even though I'm not very familiar with Ruby I can try to help.

Any thoughts @den-stripe ?

den-stripe commented 7 years ago

Hey @poislagarde ! I have not tried to run it with mongo driver >= 2.0. So not sure what is required to make it work.

thucnc commented 6 years ago

Mosql/Mongoriver doesn't work well with mongodb 3.6. It often gets below exception at line 170 of tailer.rb file (@cursor.next)

PlanExecutor killed: CappedPositionLost: CollectionScan died due to position in capped collection being deleted. Last seen record id:

So, may you advise how to upgrade mongoriver compatible with new mongodb driver ? P.S: When I rerun the mosql script after its failure, it works again without any data lost, so I tried to add a rescue block as below but it doesn't work:

     begin
        while !@stop && !state.break? && @cursor.has_next?
          state.increment
          record = @cursor.next
          case database_type
          when :mongo
            blk.call(record, state)
          when :toku
            converted = Toku.convert(record, @upstream_conn)
            converted.each do |converted_record|
              blk.call(converted_record, state)
            end
          end
        end
      rescue
        puts "exception cursor.next"
        state.increment
      end
johnnason commented 6 years ago

I'm using ->mosql->mongoriver for mongo to pg oplog streaming, and the above PR is working for me. Still doing more testing, but have streamed many many thousands of records accurately so far. Where I'd be more worried about the above PR is some connection scenarios I have may not have tried, and TokuMX.