search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.49k stars 3.02k forks source link

Managed roll out of filesystem caching support #20550

Open mosabua opened 9 months ago

mosabua commented 9 months ago

We are trying to complete the Alluxio caching support roll out now that Trino 438 is out. Following is the proposed sequence of tasks with status and mention of contributors and others helping.

Initial discussion also happened in https://trinodb.slack.com/archives/CP1MUNEUX/p1706892319598769

### Tasks
- [x] Initial merge of support for Delta Lake #18719
- [x] Merge of docs for Delta Lake #20300 in progress with @mosabua
- [x] Create and merge PR to support Iceberg , in progress with @amoghmargoor #20602
- [x] Docs PR for Iceberg, ready to go at #20599
- [x] Create PR to support Hive, https://github.com/trinodb/trino/pull/20658
- [x] Docs PR for Hive, ready to go at #20600
- [x] Merge PR for Rubix removal #20102 from @wendigo
- [ ] Create PR to support Hudi - @yihua and @codope
- [ ] Docs PR for Hudi, ready to go at #20601
- [x] Cut 439 release - under control with @martint @mosabua and @colebow
- [x] Blog about the whole effort (@mosabua) https://github.com/trinodb/trino.io/pull/615
- [x] Add tracing to the Alluxio caching - #20557 @wendigo
- [ ] https://github.com/trinodb/trino/pull/20610
- [x] Lead discussion about preferred hosts @mosabua
wendigo commented 9 months ago

I'm looking into adding tracing to Alluxio caching

mosabua commented 9 months ago

Started thread to continue the discussion around preferred hosts at https://trinodb.slack.com/archives/CP1MUNEUX/p1707517245306449

mosabua commented 9 months ago

Potentially update deps with https://github.com/trinodb/trino/pull/20685

TinoSM commented 8 months ago

AS a very random comment (please ignore if doesnt apply), do not forget to include the dependencies as part of helm chart (if that's planned https://artifacthub.io/packages/helm/trino/trino), tried to add the feature to my system (Delta/Hive connectors) and got

Caused by: java.lang.ClassNotFoundException: io.trino.hdfs.TrinoFileSystemCache$FileSystemKey

Nevermind it got fixed by updating Trino version manually instead of using the chart's default version (I expected chart and trino to be up-to-date, but it seems versions are not linked 1:1).

mosabua commented 8 months ago

@TinoSM .. the feature is part of the docker container so automatically also part of the Helm chart. Please provide more info on what you did in a separate ticket ..

mosabua commented 8 months ago

Blog post went live a while ago - here is the link https://trino.io/blog/2024/03/08/cache-refresh

Also https://www.linkedin.com/feed/update/urn:li:activity:7171977291483299840

akenO8 commented 7 months ago 
Caused by: java.lang.ClassNotFoundException: io.trino.hdfs.TrinoFileSystemCache$FileSystemKey

After I upgraded to 445, booting reported this error.

mosabua commented 6 months ago 
Caused by: java.lang.ClassNotFoundException: io.trino.hdfs.TrinoFileSystemCache$FileSystemKey

After I upgraded to 445, booting reported this error.

Please file a separate issue with details. This is not something that should happen at all .. unless yoy have a modified fork

akenO8 commented 6 months ago 
Caused by: java.lang.ClassNotFoundException: io.trino.hdfs.TrinoFileSystemCache$FileSystemKey

After I upgraded to 445, booting reported this error.

Sorry, This is an error printed when the startup of the trino-server fails due to other bugs. https://github.com/trinodb/trino/issues/21669 Please ignore it.