Add documentation for iceberg memory file system caching properties

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

https://trino.io

Apache License 2.0

10.49k stars 3.02k forks source link

Add documentation for iceberg memory file system caching properties #24061

Open sheajamba opened 2 weeks ago

sheajamba commented 2 weeks ago

Description

This PR adds a section to the Iceberg connector documentation for memory file system cache. The following properties are documented:

fs.memory-cache.ttl
fs.memory-cache.max-size
fs.memory-cache.max-content-length

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required. ( ) Release notes are required. Please propose a release note for me. ( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

sheajamba commented 2 weeks ago

@mosabua Does this need its own memory file caching page similar to file system cache?

sheajamba commented 2 weeks ago

@raunaqmorarka Can you help me fill out the default values for these properties? Thanks!

mosabua commented 2 weeks ago

@mosabua Does this need its own memory file caching page similar to file system cache?

It definitely should go into the object-storage folder and documentation since it is NOT specific to a connector. Its is part of the file system support in Trino.

However I dont know for sure how it relates to the S3, GCS, and Azure storage docs .. I think it is a separate file system support but it is also somehow related and used by all of them.. at least optionally.

The memory-cache config stuff is only part of the overall memory file system .. we might have to collect all that info from @electrum and @raunaqmorarka and the code and figure out how it all hangs together and how to best document that.

raunaqmorarka commented 4 days ago

We definitely need to understand this better and put it into the object storage section.. details to be determined.

I think this is documented in the right place. Although the implementation is written to be connector agnostic, it is used only within iceberg connector at the moment and the fact that it is a "memory filesystem cache" is an implementation detail.

mosabua commented 3 days ago

We definitely need to understand this better and put it into the object storage section.. details to be determined.

I think this is documented in the right place. Although the implementation is written to be connector agnostic, it is used only within iceberg connector at the moment and the fact that it is a "memory filesystem cache" is an implementation detail.

.. how do we make sure we dont forget when we add it to another connector. Also .. are all properties covered now? I would want to make sure it is at least in a separate fragment file, with a generic filename and content, and not in the connector docs file itself.