Use one stream for all rebasers

nickgerace commented 1 week ago

Description

This PR uses one stream for all rebasers and reduces the number of consumers per change set by 50% (from two to one). This should alleviate some pressure on NATS Jetstream and dramatically reduce the total number of streams (i.e. one stream per change set to one stream for all rebasers).

Additional Changes

Remove rebaser-core because it was entirely unused
Make rebaser server's from_services setup async
Make the rebaser server own the stream that it uses to create consumers
Expose external APIs into the layer cache that should eventually move out
Delete unused shutdown handle for the rebaser server

Disclaimer

This PR contains an additional commit that should be reverted later. Please read the commit's description for more information.

nickgerace commented 6 days ago

@johnrwatson @sprutton1 @fnichol this is ready, but we need to talk about migration and deployment. The PR might need changes, the real world might need to change, or a little of both.

nickgerace commented 6 days ago

Merging since the plan is set and this code works for it.

systeminit / si