Handle transient network failures and support custom staging dir path

[ryangsteele]

This PR makes two improvements:

1. If a transient connectivity issue occurs while rsyncing a batch from the remote host, sync.py will wait for 5 seconds and then retry a sync of that batch. If, after 3 such attempts, syncing has not succeeded, the metrics in that failed batch are saved to a file so that they can be easily retried by the user at a later time.

2. Adds a second command-line option (-t or --tmpdir) which allows the user to specify where the temporary rsync staging dirs will be created. This can be useful for very large syncs where the filesystem containing /tmp (or wherever mkdtemp would create it on the user's OS) isn't big enough to store all the synced metrics.

[deniszh]

Hello @ryangsteele ,

Thanks for your PR! I like the idea, but left couple of comments, please check.

[ryangsteele]

Hi @deniszh!

Thanks for creating such a useful tool, and for reviewing my PR so promptly. I believe all of your concerns have been addressed. Mind taking another look when you have a moment?

[ryangsteele]

I realized I forgot to update the README.md with the new options... I'll do that now. Sorry.

[deniszh]

Ah, right. Please do so.

[ryangsteele]

There we go - all set! :)

[deniszh]

@ryangsteele : I'm sorry to say that - but tests are broken. :(

[ryangsteele]

Sorry for the back-and-forth there, @deniszh, that's on me. The checks are passing in my local dev env at this time.

[deniszh]

Now tests are green in Travis too! Merging.

[ryangsteele]

Thank you!