search

stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.63k stars 225 forks source link

Rate limiting? #61

Closed AndreaCrotti closed 7 years ago

AndreaCrotti commented 10 years ago

I've tried to run a migration today on production and it's really fast, I got

550 k rows in 2 minutes, which is around 5k per second. However that's too fast, because the app suffered while I was doing that! In verbose mode it did less than 3k per second and that didn't really affect the db that much..

Is there a way to do some rate limiting to make it run a bit slower (even a bit hacky changing the code)? Even introducing a silly sleep here and there if that makes the trick would be enough for now..

nelhage commented 10 years ago

Do you run a replica set in production, so that you can do an import from a secondary? That's our usual recommendation to avoid performance degradation.

But we could probably add a rate limit or throttle. If you want to experiment, adding something in handle_op would probably be the right place: https://github.com/stripe/mosql/blob/master/lib/mosql/streamer.rb#L195

AndreaCrotti commented 10 years ago

Yes well the Mongo database is pretty fast and it does not have problems, it's just that it's writing too fast in Postgres and thus causing a very high load there.

I think I'll end up just adding some sleeps here and there, not a great solution but it might work at least..

AndreaCrotti commented 7 years ago

I guess we can close this by now