sum-up: things older java versions are blocking

[hpvd]

To get an overview which things we are missing with staying using/being compatible to older Java versions, and determine the right point of time when its maybe worth to drop one or find a way to work around it, I like to start this list.

needs java 21 (LTS):

• Lucene v10, https://lucene.apache.org/core/10_0_0/changes/Changes.html highlights from https://lucene.apache.org/core/corenews.html#apache-lucenetm-1000-available
  • A new IndexInput#prefetch API has been added, allowing query evaluation logic to let the Directory know about regions of data that are about to be read. This helps perform I/O concurrently under the hood. MMapDirectory implements this API using the madvise system call and the MADV_WILLNEED flag on Linux and Mac OS.
  • Lucene now supports sparse indexing on doc values via FieldType#setDocValuesSkipIndexType. The sparse index will record the minimum and maximum values per block of doc IDs. Used in conjunction with index sorting to cluster similar documents together, this allows for very space-efficient and CPU-efficient filtering.
  • Search concurrency is now decoupled from the index geometry, so that an index can be searched using any number of threads, regardless of its number of segments.
  • Kmeans clustering on vectors

issue to upgrade lucene to v10.x https://github.com/apache/pinot/issues/14354

needs java 22:

• Use Foreign Memory API, see https://github.com/apache/pinot/issues/12809

Please add your knowledge/findings to the comments so we can update this list. Maybe we can also highlighting some details why supporting it may be worth it (beside always happening bug and security fixes)

[gortiz]

You can add https://github.com/apache/pinot/issues/12809 (although it requires Java 22).

IMHO the target should not be to drop Java < 21 support but start using multi-release jars to be able to provide features that are only available in specific runtimes.

[gortiz]

Also notice that it may be acceptable to increase the minimum Java runtime version for Pinot processes (servers, brokers, controllers and minions), but increasing the minimum version of the Pinot driver is more sensible.

The Pinot driver build process is pretty coupled with Pinot processes. Due to the way the driver is being built, it depends on some core projects of Pinot processes. We in fact compile these modules twice: Once with Java 11 and one with Java 8. Given that these two Java versions are very close in terms of syntax there is usually no problem. If we end up moving the services code to Java 21 that won't be the case.

So any plan to migrate Pinot services to a newer version would need to analyze the impact in the driver. Probably we would need to split the driver code from the services code or, in a more ambitious way, try to implement a third party protocol (mysql or postgres) and use their drivers.