[CORE-7750] PoC - Creating topic with huge number of partitions leads to segfault

[IoannisRP]

Fixes: CORE-7750

The size of the allocation_request created as part of the partition allocation scales linearly with the number of partitions needed to be created. Currently, the check to see if the request can be accepted happens on the allocation_request object. When the number of requested partitions is too big, we run out of memory while creating this object.

The fix is to be lazy on the creation of the allocation_request, where possible, and do the first step of the validation before we create the full object.

TODO:

• use lazy path in topics_frontend::do_create_partition.

Backports Required

• [ ] none - not a bug fix
• [ ] none - this is a backport
• [ ] none - issue does not exist in previous branches
• [ ] none - papercut/not impactful enough to backport
• [x] v24.2.x
• [ ] v24.1.x
• [ ] v23.3.x

Release Notes

Bug Fixes

• TODO

[IoannisRP]

@rockwotj good points. I'll address them and i'll add the do_create_partion lazy path.

[michael-redpanda]

/dt