search

facebookarchive / rocks-strata

Other
133 stars 24 forks source link

Limiting backup speed #3

Closed jonathannorris closed 8 years ago

jonathannorris commented 8 years ago

Hey rocks team, first off thanks for all the great work you have put into mongo rocks and this awesome backup tool. We've been running rocks in our mongo replica sets for a couple weeks so far, and are impressed with the speed and stability over Wired Tiger.

The main issue we are seeing is that the strata backups are too fast, and maxing out the network bandwidth on our AWS instances. This causes some latency spikes and slowing of mongo replication. What would be the best way to limit the number of connections strata makes to S3 and slowing the speed of the backups to not max out the network connection?

We could probably put together a PR with it, but some advice on how to implement it would be great.

screenshot 2016-01-15 11 44 14

tredman commented 8 years ago

Hey! Thanks for trying out MongoRocks.

Have you tried setting the --max-background-copies flag? The default is 16 - which configures strata to use 16 threads to write to S3. You can adjust this up or down depending on your available bandwidth and I/O. I recommend reducing this until your bandwidth is no longer saturated.

jonathannorris commented 8 years ago

Thanks for pointing that out @tredman! Didn't notice it on my first look. We will give that a shot.