Prototype and test relational database approach to the state store

[patchwork01]

Background

See related issues:

• https://github.com/gchq/sleeper/issues/1355
• https://github.com/gchq/sleeper/issues/1700

Description

See design document.

We'd like to know whether the relational database approach to the state store can improve on the consistency issues we've had querying the whole state from the DynamoDB state store at once.

Analysis

A prototype version of this could just support adding files to the state store and performing compaction on those files. We could just support the ParallelCompactionsIT system test, and see how far we can increase the parallelism of compactions.

We could split out issues to:

• Implement those operations against PostgreSQL in an integration test
• Deploy the required resources with the CDK to run the parallel compactions system test with this state store

[gaffer01]

Design document updated October 10th 2024 based on initial experimentation with a PostgreSQL state store.