[Python] OSError: Unable to load libjvm: /usr/java/latest//lib/amd64/server/libjvm.so

[xiongma]

from pyarrow import fs

hdfs, _ = fs.FileSystem.from_uri('hdfs://haruna')

when i use this type of code to create a connect, it raise this error

    hdfs, _ = fs.FileSystem.from_uri('hdfs://haruna')
  File "pyarrow/_fs.pyx", line 463, in pyarrow._fs.FileSystem.from_uri
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Unable to load libjvm: /usr/java/latest//lib/amd64/server/libjvm.so: cannot open shared object file: No such file or directory

environments

export JAVA_HOME=/opt/tiger/jdk/jdk8u265-b01
export HADOOP_HOME=/opt/tiger/yarn_deploy/hadoop
export CLASSPATH=`/opt/tiger/yarn_deploy/hadoop/bin/hadoop classpath --glob`
export LD_LIBRARY_PATH=/opt/tiger/yarn_deploy/hadoop/lib/native/
export LIBHDFS_OPTS='-Dhadoop.root.logger=INFO,console'

[kou]

Could you show the output of find /opt/tiger/jdk/jdk8u265-b01 -name 'libjvm.so'?

[Hayder-Aziz-cardano]

same issue here. Its not picking up the correct libjvm location despite java_home being correctly set.

also its linking to a location with double // between latest and lib above.

[kou]

Could you show the output of find ${JAVA_HOME} -name 'libjvm.so'?

[Hayder-Aziz-cardano]

/usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so

[kou]

echo ${JAVA_HOME}

[kou]

ls -lah /usr/lib/jvm/

[Hayder-Aziz-cardano]

echo ${JAVA_HOME} /usr/lib/jvm/java-17-openjdk-amd64/lib/

ls -lah /usr/lib/jvm/ total 16K drwxr-xr-x 3 root root 4.0K Jan 27 11:42 . drwxr-xr-x 71 root root 4.0K Jan 27 11:42 .. -rw-r--r-- 1 root root 1.8K Oct 24 13:44 .java-1.17.0-openjdk-amd64.jinfo lrwxrwxrwx 1 root root 21 Oct 24 13:44 java-1.17.0-openjdk-amd64 -> java-17-openjdk-amd64 drwxr-xr-x 7 root root 4.0K Jan 27 11:42 java-17-openjdk-amd64

[kou]

Could you try export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64?