Multi-partition transaction coordinator topic

[rystsov]

Who is this for and what problem do they have today?

Transactional coordinator is a bottleneck for the transactional processing, if we partition it similar to the consumer groups we'll reduce contention for the partitions and increase performance.

What are the success criteria?

Increased performance. In-place upgrade of existing clusters from single partition--> multi-partition.

Why is solving this problem impactful?

It's nice to be the fastest queue in the west

Additional notes

TBD

JIRA Link: CORE-1006

[mattschumpert]

Moved this to committed since it's already part of 23.2 and the upgrade handling should be clearly a committed item for Q2. @vshtokman

[toddfarmer]

@mattschumpert: Just for clarity, does your latest comment mean that the work proposed here needs to be done prior to GA of v23.2?

[mattschumpert]

@toddfarmer Yes, and it's well underway, we should link the PRs here @vshtokman

[vshtokman]

I believe this is a duplicate of https://github.com/redpanda-data/planning/issues/145.

The static component (sharding the coordinator topic for new clusters) has been merged into dev.

The dynamic component (sharding the coordinator topic for existing clusters) is currently on hold.

The project has been tracked using the project board. Once the project resumes, I expect us to continue using the board.

[mattschumpert]

@vshtokman , @piyushredpanda wanted to keep this in view for 23.3 because the in-place migration for existing clusters is still outstanding work post 23.2, hence we marked 2 weeks of spillover for this against this. Fine if we want to fork this to a new ticket which might be more accurate but we wanted to account for it somehow