search

apache / polaris

Apache Polaris, the interoperable, open source catalog for Apache Iceberg
https://polaris.apache.org/
Apache License 2.0
1.17k stars 130 forks source link

Use VirtualThreads for Polaris TaskExecutor #411

Open jbonofre opened 3 weeks ago

jbonofre commented 3 weeks ago

VirtualThreads have been added in JDK19+. It's good improvement on "regular" threads. The Polaris TaskExecutor is using regular threads for now. Using virtual threads should provide a performance boost.

snazy commented 3 weeks ago

IMHO we should be really careful with Java virtual threads, because those are not a "just turn it on and everything's faster & better" thing. VTs come with quite some fine-print. In a nutshell: everything that runs on a VT must meet a bunch of criteria - otherwise the stability of the whole process is at risk, likely not in every environment ("number of CPUs").

Few posts that discuss this topic:

eric-maynard commented 3 weeks ago

Agreed that we should look into optimizing this, though I don't know enough about virtual threads to have a strong opinion about the implementation. @snazy's concerns above seem valid.

Long-term, I am hoping that @flyrain's recent proposal will let us fully delegate this kind of work to an external service and make this problem moot.