Test mode to randomly return 503 temporary errors

[tleyden]

This should be done at the bucket level, which will cover both BLIP and REST APIs

• [x] Enhance LeakyBucket to support the ability to short-circuit retry loop and return errors on every nth bucket operation (will probably need to add a counter)
• [x] Verify that the temp errors are correctly propagated in API
• [x] Expose LeakyBucket config to SG JSON config
• [x] Update AsCouchbaseBucket to check for LeakyBucket
• [x] Run LiteCore integration tests against SG configured with injected failures, verify that it still works
• [x] Run sg-replicate to verify it retries all 503 errors

Originally from https://github.com/couchbase/sync_gateway/issues/3509#issuecomment-388919848:

Should evaluate how to easily test client behaviour without needing to generate significant load.

[tleyden]

Run LiteCore integration tests against SG configured with injected failures, verify that it still works

The initial run is showing an issue (and possibly more):

https://github.com/couchbase/couchbase-lite-core/issues/525

[tleyden]

Run sg-replicate to verify it retries all 503 errors

I can see by inspecting the code that there are places where it will abort the replication if it receives a 503 error. In this case, fetching the remote checkpoint.

https://github.com/couchbaselabs/sg-replicate/blob/cc411eae87827555f492df4c5282fc6684693584/synctube.go#L221-L226

} else if resp.StatusCode >= 400 {
        // we got an error, lets abort
        r.LogTo("Replicate", "4xx error(not 404) getting checkpoint")
        event := NewReplicationEvent(FETCH_CHECKPOINT_FAILED)
        r.sendEventWithTimeout(event)
    } else if ...  {