Open ChrisHegarty opened 3 months ago
Pinging @elastic/es-search (Team:Search)
More repo steps:
// build localDistro as a convenience to get a local JDK and build the native libs, e.g.
./gradlew localDistro
export RUNTIME_JAVA_HOME=/home/ubuntu/git/elasticsearch/build/distribution/local/elasticsearch-8.14.0-SNAPSHOT/jdk/
// enable logging, e.g.
$ git diff benchmarks/src/main/resources/log4j2.properties
@@ -4,5 +4,5 @@ appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c] [%node_name]%marker %m%n
# Do not log at all if it is not really critical - we're in a benchmark
-rootLogger.level = error
+rootLogger.level = debug
rootLogger.appenderRef.console.ref = console
// run the benchmark
./gradlew -p benchmarks run --args "org.elasticsearch.benchmark.vector.VectorScorerBenchmark -p dims=1024"
...
[2024-05-07T14:04:49,404][WARN ][org.elasticsearch.nativeaccess.NativeAccess] [
ode_name] Unable to load native provider. Native methods will be disabled.
java.lang.UnsatisfiedLinkError: Could not find libsystemd.so.0 in java.library.path: /home/ubuntu/git/elasticsearch/libs/native/libraries/build/platform/linux-aarch64
at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.findLibSystemd(JdkSystemdLibrary.java:53) ~[elasticsearch-native-8.14.0-SNAPSHOT.jar:8.14.0-SNAPSHOT]
at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.<clinit>(JdkSystemdLibrary.java:29) ~[elasticsearch-native-8.14.0-SNAPSHOT.jar:8.14.0-SNAPSHOT]
...
Pinging @elastic/es-search-relevance (Team:Search Relevance)
When adding the native int7 vector scorer a benchmark was also added. The benchmark requires the
java.library.path
to the set so that it can locate the native libraries. Not just thelibvec.so
, but alsolibc
,systemd
, etc, since they are required once the Elasticsearch native module is initialised.The problem is that the benchmark does not correctly add the library path (nor does it depend upon the native module, which it should), it just does this (in benchmarks/build.gradle):
, which is not sufficient for finding libc or systemd on some Linux distributions. Consequently, the benchmark fails to run. Unless one was to patch it with something like this:
For testing, the native library is configured in ElasticsearchJavaBasePlugin, through a
TestUtil.getTestLibraryPath
. Would be great if we could modify the benchmark'srun
task to have the library path set correctly, or somehow useTestUtil.getTestLibraryPath
directly.To reproduce the problem, just run the benchmark on Linux, say Ubuntu (aarch64 for now, until x64 support is added):