search

stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.62k stars 224 forks source link

Inconsistent replication #19

Closed mrgordon closed 11 years ago

mrgordon commented 11 years ago

Have you seen or heard any accounts of inconsistent data? Our records don't line up precisely between Mongo and PostgreSQL. Some records don't get replicated at all despite trying to run the sync from scratch. We also see duplicate inserts which are blocked by the unique constraint on the id column in the PostgreSQL logs.

Happy to provide any info that would be helpful.

nelhage commented 11 years ago

Hi Matthew,

Do you have logs from the MoSQL import? Are there any lines at WARN priority? (you can just grep for WARN). At the moment, if MoSQL encounters records that don't match the declared schema, it will ignore them and issue a WARN-level log statement.

The duplicate inserts are harmless -- MoSQL implements "upsert" by doing an INSERT and catching the duplicate-key exception and doing an UPDATE instead. I actually just pushed a change today that reworks this logic to avoid the exceptions in the Postgres logs, since they're noisy and worrying if you don't know to expect them.

nelhage commented 11 years ago

Hi,

I haven't heard any reply in about a month, so I'm going to close this issue. Please feel free to re-open if you're still seeing problems.