Add Caching

[c-thiel]

Currently we issue a lot of DB requests. In many places, we would benefit greatly from Caching. A bit of motivation:

• signing requests typically sign requests for many files at once in very short timeframes
• Table-Identifier to ID conversion is required by AuthZ and typically called multiple times, as clients first create and then commit multiple changes to a table.
• Signing depends on a very expensive query: Searching for a query based on its location - the location of a table is almost static.

Good places to implement caching serve slowly changing data and are queried often. However, we must always assume that we run in a distributed environment. Thus, we should never cache the full TableMetadata that can change frequently for a table. In contrast, location & uuid can be considered slowly changing.

Some calls that I think should be cached:

• table_ident_to_id: https://github.com/hansetag/iceberg-rest-server/blob/1c39f69d205a3275b5f1b5d50311954c8756869b/crates/iceberg-rest-server/src/catalog/tables.rs#L200
• table_idents_to_ids: https://github.com/hansetag/iceberg-rest-server/blob/1c39f69d205a3275b5f1b5d50311954c8756869b/crates/iceberg-rest-server/src/catalog/tables.rs#L615
• get_table_metadata_by_s3_location: https://github.com/hansetag/iceberg-rest-server/blob/1c39f69d205a3275b5f1b5d50311954c8756869b/crates/iceberg-rest-server/src/catalog/s3_signer.rs#L69
• get_table_metadata_by_id: https://github.com/hansetag/iceberg-rest-server/blob/1c39f69d205a3275b5f1b5d50311954c8756869b/crates/iceberg-rest-server/src/catalog/s3_signer.rs#L115 - while on it, maybe rename get_table_metadata_by_id to something else, to make it clearer that this function is not loading TableMetadata object, but only Metadata about the get_table_metadata_by_id?

I am not sure yet what would be the best place to add caching. Ideally, its not implementation specific, so it shouldn't be part of postgres, so not in this folder: https://github.com/hansetag/iceberg-rest-server/tree/main/crates/iceberg-rest-server/src/implementations/postgres

We also have to make the choice of distributed vs. local caching. I currently prefer local caching due to less dependencies. Postgres and most other DBs cache as well.

[twuebi]

What are your thoughts regarding cache invalidation? Tables may be dropped or staged tables may be overwritten, in both cases the cached name -> uuid mapping becomes stale.

I can see a cache work for the location mapping, here we could simply store the full location based on previous lookups, if this gets stale we still have the same behavior, a 404 upon fetching from pg using the full location. We'd get rid of the prefix based lookup this way.

For all other caches, I only see them working with some fallback logic that retries and cleans the cache executed from the respective handler fn which sounds boilerplatey.