Avoiding the use of scala's Array and ArrayBuffer because they require the use of ClassTag, which requires extra permissions to be used in Elasticsearch. These permissions have been enabled for a while now, and I went down the rabbithole of potentially adding more for Scala 3, so I'd rather get rid of them now that I know how.
Specifically:
changing vector parsing from using ArrayBuffer to using custom specialized Java implementations. Also added a trick to set the array buffer's initial array capacity to be the final capacity of the previous array. This exploits the fact that we're usually indexing and querying vectors of the same length. This is about ~2.8x faster than using a scala ArrayBuffer initialized to the final size, and ~3.3x faster than using a scala ArrayBuffer initialized to the default size. But this only shows up in a microbenchmark. It doesn't show up in the ann-benchmarks as JSON deserialization is not really a bottleneck.
changing some plugin internals that were mapping and flatMapping Arrays to instead use foreach with ListBuffers. This avoids the invocation of methods that require a ClassTag.
Related Issue
393
Changes
Avoiding the use of scala's Array and ArrayBuffer because they require the use of ClassTag, which requires extra permissions to be used in Elasticsearch. These permissions have been enabled for a while now, and I went down the rabbithole of potentially adding more for Scala 3, so I'd rather get rid of them now that I know how.
Specifically:
Testing and Validation
Standard CI and benchmarking.