Closed stanlemon closed 6 years ago
I know you have a doc somewhere, but it may be helpful to updatethe PR description with clear definitions of all the states and what they mean.
This PR will need a README overhaul, I'd like to work on that in a separate PR.
I want to add the validation mentioned by StevieP but I'd like to do it in a separate PR as part of the work on #75 as I think it'll be easier to address when that interface allows saving partitions together.
let er rip.
Summary
This PR adds a new state to the sideline sequence which we are calling resume.
Current States
Today sidelines are binary, they can be started or stopped. Starting a sideline means that a starting offset is captured for a future virtual spout and a filter is applied to the firehose virtual spout. This is typically done around the concept of a 'tenant', for example you might want to sideline account Foobar for a period of time. Stopping a sideline means that an ending offset is captured for a virtual spout that will be now created using the starting offset saved during the start of the sideline. That new virtual spout is immediately added to the coordinator with a filter that only emits the tenant in question and the filter is removed altogether from the firehose.
Potential Problems
The consequences of the binary approach are that ordering gets wildly thrown off and when a sideline stops we are essentially doubling a tenants emitted messages. This can be controlled with a different MessageBuffer, but the underlying problem remains: if a tenant's operations required sidelining we go from 0-60 (or 65) almost immediately without slowing down the emission of data into the main spout.
New States
Moving forward, and the goal of this PR is to split up the current stopping state into two different states:
Why does this matter?
The primary use we want to support here is throttling of traffic for a single tenant without having to know the ending point. So by using this new 'resume' state and a MessageBuffer that emits at a slower rate for sidelines, someone could slow the emission of a tenant into the underlying
DynamicSpout
, and then at the point at which they are ready to resume full processing they can set that sideline to 'resolve'.It's worth noting here that by calling resume() and resolve() back to back you should have the functional equivalent of today's stop() and that nothing is changing with the current start() process.
In the Future
Related to this, my suspicion is that this implementation will also give us better mechanics to control the timing of resolving the sideline so that some day in the future we can try to target the sideline once it's near the tail of the firehose (probably using some tolerance level), but that's not the purpose of this PR just an additional thought on why this change makes sense.