Will Trino S3 file system support AWS Common Runtime (CRT)

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

https://trino.io

Apache License 2.0

10.46k stars 3.01k forks source link

Will Trino S3 file system support AWS Common Runtime (CRT) #23855

Open kekwan opened 3 weeks ago

kekwan commented 3 weeks ago

We've seen some articles stating that AWS CRT is improving S3 throughput drastically.

S3 CRT does not look enabled as of latest Trino version. Are there plans to support or add this in the future? I see https://github.com/trinodb/trino/pull/17522#discussion_r1278664530 from @electrum from an old PR saying we don't want to use it. Is that still the case?

wendigo commented 3 weeks ago

This should be now possible since we've moved to JDK 22 which solved the pinning issue in G1 GC. Any thoughts @electrum ?

electrum commented 3 weeks ago

It looks like the linked benchmark is for Python. Have you seen any benchmarks for Java? If someone wants to try this out in Trino and run a benchmark, this would help inform if this is something we want to support.

wendigo commented 3 weeks ago

https://github.com/trinodb/trino/pull/23875