Add `capDelay` to the retry policies

[nhibberd]

• 60 seconds should be a good starting point http://hackage.haskell.org/package/retry-0.6/docs/Control-Retry.html#v:capDelay

[michaelneale]

I think it is hard to be generic about backoffs - s3 as part of intra region pipeline probably could be shorter than inter region, for example. However, usually backoffs being larger pays off nicely (even though it may seem you are introducing a lot of latency). I guess some tuning will be required, trial and error (which what nick has been doing).

[charleso]

https://github.com/ambiata/mismi/pull/147

[olorin]

@michaelneale I'm not sure if network delays would be a significant factor - on the scale of tens of seconds inter-region latency should be tiny, and TCP should be able to cope with the tubes being clogged. I agree with trial and error though.

[michaelneale]

I was thinking also non AWS users - but still, yes it may not matter. I think 60 seconds is a bit low, but this is s3, not some misc. api on the internet (most monitoring systems have a resolution of a minute to do anything much, so if it was back pressure due to failure, more time may be needed, however this seems to be rate limiting, so carry on!). On the one hand we have a knob to turn up called retries - on the other hand, library consumers that are baffled at sudden changes in times that things take to run (unless they turn on tracing/logging).