search

HydrologicEngineeringCenter / Vortex

data processing utilities
MIT License
25 stars 7 forks source link

Vortex Jython API in Linux #94

Closed jlgutenson closed 11 months ago

jlgutenson commented 11 months ago

Hey @tombrauer and Co.,

Sorry to trouble you guys again but I suspect I have an issue that you have a quick solution for.

I'm trying to use Vortex on a Linux (Ubuntu) machine through Jython (clunky I know).

I'm having trouble getting Jython to recognize the Vortex library.

Here's the Jython command I'm running. This is basically a repeat of the Windows command, just formatted for Linux.

jython -Djava.library.path=$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal met_data_import.py

Do you know how to get Jython to recognize the Vortex library on a Linux machine?

Thanks in advance, Joseph

tombrauer commented 11 months ago

Which version of vortex? When I migrated to Java 17 (v0.11.1) I had to update the configuration a bit to use jython-standalone.jar instead of jython. See documentation here.

jlgutenson commented 11 months ago

I'm using Vortex 0.11.0. I think that the latest version you have set up for Linux. However, the Vortex jar file is labeled "vortex-0.12.0-dev.0.uncommitted+b0bd022.jar" in the lib directory.

tombrauer commented 11 months ago

@jlgutenson can you post the specific error message that you are getting?

jlgutenson commented 11 months ago

Hey @tombrauer,

I tried your method here, just Linux-ized it. Here's my .sh file for reference.

vortex_2023081710.zip

Here's the error I recieved in this instance. Error: Could not find or load main class org.python.util.jython Caused by: java.lang.ClassNotFoundException: org.python.util.jython

Previously, the error I was receiving was that Jython could find any of the HEC libraries when I attempt to import them in Jython.

Thanks, Joseph

jlgutenson commented 11 months ago

Update, @tombrauer , you can disregard the comment above . I needed to add "export" in front of the variables I set in bash script. Here's the bash script.

vortex_2023081711.zip

I've now progressed to the error where when running myImport.process() in the Jython script, I receive this error from Java

image

My guess is I'm doing something fairly simple incorrectly.

This is the Vortex version I'm running (Linux edition).

tombrauer commented 11 months ago

UnsatisfiedLinkError means a JNI library isn't linking. Do you have a -Djava.library.path=<path to javaHeclib.so>:<path to gdal> arg?

tombrauer commented 11 months ago

You could try calling this shell script from yours to configure the environment, or take bits and pieces to configure your own: https://github.com/HydrologicEngineeringCenter/Vortex/blob/master/vortex-ui/package/linux/run-vortex.sh

This is how we set the environment for Linux.

jlgutenson commented 11 months ago

There's a /usr/lib/jni path being passed in here that I don't have on my machine.

Maybe that's the issue?

I have the -Djava.library.path=<path to javaHeclib.so>:<path to gdal> set but not that third path.

jlgutenson commented 11 months ago

I've still been hitting a brick wall with the UnsatisfiedLinkError. ChatGPT suggested setting the LD_LIBRARY_PATH in the environment, similar to how you set PATH in your environment.

Here's my current .sh file setup:

VORTEX_HOME="/home/jlgutenson/rtspy/vortex-0.11.0"
export PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal:$PATH"
export GDAL_DATA="$VORTEX_HOME/bin/gdal/gdal-data"
export LD_LIBRARY_PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal:$LD_LIBRARY_PATH"
export PROJ_LIB="$VORTEX_HOME/bin/gdal/proj"
JAVA_VORTEX_PATH="$VORTEX_HOME/jre/bin/java"
JAVA_VORTEX_LIB_PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal"
export CLASSPATH="/home/jlgutenson/rtspy/jython/jython-standalone-2.7.3.jar:$VORTEX_HOME/lib/*"
$JAVA_VORTEX_PATH -Xmx12g -Djava.library.path="$JAVA_VORTEX_LIB_PATH" org.python.util.jython met_data_import.py "/mnt/d/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/Scripts/build_hms_inputs/hrrr_subhourly/20230818/16" "/mnt/d/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/1.2.2.2.2/Models/HEC_HMS_411beta16/1_RVD/1_HEC-HMS_Model/RVD_NAD8310171/gis/RVD_83_1/RVD_83_1.shp" "/mnt/c/Users/Joseph Gutenson/Desktop/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/1.2.2.2.2/Models/HEC_HMS_411beta16/1. RVD/1. HEC-HMS Model/RVD_NAD8310171/RVDJune2018_JLG_scripted_1.dss" "Total_precipitation_surface_15_Minute_Accumulation" "HRRR"

When I try setting LD_LIBRARY_PATH, I end up with this error:

Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!

And separating out the LD_LIBRARY_PATH, it seems like something funky is going on between my system and GDAL.

jlgutenson commented 11 months ago

I've built the program on my local Linux machine with the source code and Gradle. I've had the same error returned to me that the 0.11.0 precompiled version was returning to me.

I get two different errors by two different means.

If I point -Djava.library.path to JAVA_VORTEX_LIB_PATH I get a delightfully unspecific UnsatisfiedLinkError.

If I use the LD_LIBRARY_PATH to set my library. I get the error: Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!. Point the environment to the "/bin/gdal/" directory seems to cause this error.

This is all being ran in a bare bones Anaconda environment that is using Python version 3.8 and has OpenJDK and it's dependencies installed. I've tried versions 8, 11, 17, and 20 of OpenJDK to see if that's the issue. The current master branch of Vortex source code is working for 17, right? JDK version 20 worked properly on Windows.

tombrauer commented 11 months ago

@jlgutenson I would try using Ubuntu 20.04 for your Linux version if you have not already. That is what we've tested on.

jlgutenson commented 11 months ago

Thanks for the suggestion @tombrauer!

When I tried using Ubuntu 20.04, I got the same result.

How sensitive is Vortex to Java versions? I'm using 17.0.3.

tombrauer commented 11 months ago

The source code is compiled in Java 17 so you should be good with Java 17 or newer.

jlgutenson commented 11 months ago

10-4, just wanted to verify.

Moving to Ubuntu 20.04 did seem to solve the Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed! error.

Now I'm just getting the "java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError" with or without using LD_LIBRARY_PATH.

The gbx9 and ncx4 files are being written, so I know Vortex is at least opening up the grib2 files.

tombrauer commented 11 months ago

I was going to ask where exactly in your script things are failing but I'm guessing it will be troublesome for you to debug that. I'd say it's highly likely you're not linking to GDAL with a possibility that you are not linking to javaHeclib. GDAL should be bundled with the Linux distribution meaning that you shouldn't have to do a manual install. The GDAL version should be consistent with the jar version used in Vortex, i.e. GDAL 3.2.*. If the GDAL natives do not align with the major and minor version number, I would expect an unsatisfied link error. For GDAL you must have the JNI lib specified as a -Djava.library.path arg, and you must have GDAL itself on LD_LIBRARY_PATH. Check paths to verify libraries are where you say they are.

jlgutenson commented 11 months ago

Something weird to at least pass along is that Vortex ran when I accidently pointed it at a directory of compressed grib2.gz files. It's when I point Vortex at a directory of grib2 files that I get the "java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError".

tombrauer commented 11 months ago

Does it produce meaningful output when point at gzipped files?

jlgutenson commented 11 months ago

It does and DSSVue can read it. However, the time stamps are all messed up in terms of days and time step (e.g., it should generate a DSS file that ends on 8/21 and has a 1 hour time step but instead generates a DSS file that ends on 8/11 and has a 15 minute time step).

Clipping down to the watershed appears to work correctly as well.

jlgutenson commented 11 months ago

Here's a little more context, looks like something is going haywire with UCAR's Grib2Iosp function. I got this error when I manually pointed Java to the CLASSPATH.

image

nickvan612 commented 11 months ago

Hi @jlgutenson, for the UnsatisfiedLinkError you encountered below:

Something weird to at least pass along is that Vortex ran when I accidently pointed it at a directory of compressed grib2.gz files. It's when I point Vortex at a directory of grib2 files that I get the "java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError".

Would you help me screenshot the error for more details? Thanks! :grin: 👍

jlgutenson commented 11 months ago

Hey @nickvan612, thanks for taking a look! Here you go. See below for the UnsatisifiedLinkError:

image

Manually providing Java the classpath revealed that something seems off with how the Grib2 is being handled by Vortex. The error below directly proceeds the error above:

image

tombrauer commented 11 months ago

Creating the HEC-HMS control file. is really odd. Is that something in your code?

jlgutenson commented 11 months ago

Creating the HEC-HMS control file. is really odd. Is that something in your code?

Yes, sorry, that's my code.

tombrauer commented 11 months ago

Nothing in the second stack is in vortex. It's all in the ucar.nc2 dependency 🤔

nickvan612 commented 11 months ago

Hi @jlgutenson, I was looking for the part under UnsatisfiedLinkError :grin: 👍

jlgutenson commented 11 months ago

Yeah, this looks like some sort of issue with the ucar.nc2 library that Vortex is using. That may explain why I can interact with .gz files but not the .grib2 files.

here's how I'm currently setting up my batch runs...

VORTEX_HOME="/home/jlgutenson/vortex-0.11.0"
export PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal:$PATH"
export GDAL_DATA="$VORTEX_HOME/bin/gdal/gdal-data"
export PROJ_LIB="$VORTEX_HOME/bin/gdal/proj"
JAVA_VORTEX_PATH="$VORTEX_HOME/jre/bin/java"
JAVA_VORTEX_LIB_PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal"
CLASS_PATH="/home/jlgutenson/jython-standalone-2.7.3.jar:$VORTEX_HOME/lib/*"
export LD_LIBRARY_PATH="$VORTEX_HOME/bin/gdal"
$JAVA_VORTEX_PATH -Xmx12g -Djava.library.path="$JAVA_VORTEX_LIB_PATH" -classpath "$CLASS_PATH" org.python.util.jython met_data_import.py "/mnt/d/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/Scripts/build_hms_inputs/mrms_pass2/2023082123" "/mnt/d/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/1.2.2.2.2/Models/HEC_HMS_411beta16/1_RVD/1_HEC-HMS_Model/RVD_NAD8310171/gis/RVD_83_1/RVD_83_1.shp" "/mnt/c/Users/Joseph Gutenson/Desktop/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/1.2.2.2.2/Models/HEC_HMS_411beta16/1. RVD/1. HEC-HMS Model/RVD_NAD8310171/RVDJune2018_JLG_scripted_1.dss" "MultiSensor_QPE_01H_Pass2_altitude_above_msl" "MRMS"

also, here are the .gz files I mentioned.

2023082120.zip

jlgutenson commented 11 months ago

This morning, I ran gradle test on the main repo's code. I received 66 failed tests. See the attached.

test.zip

Looks like a lot of these originate with GDAL. So now I guess I'm back to blaming GDAL 🤣.

I notice in the build.gradle.kts file that the Linux install is being pointed at 3.0.4. But I think @tombrauer mentioned that Vortex requires GDAL version 3.2.*. Maybe that's the culprit?

Anyway, I'm very green with Java/Gradle/etc. and tried to point the build.gradle.kts file to GDAL version 3.2.0 but can't seem to get that to work. Gradle can't find a Maven repo that contains that tar.gz file. Is that something you guys are storing on the HEC repository?

tombrauer commented 11 months ago

You could test by removing the GDAL 3.2. jar from the Vortex distribution and replacing with a GDAL 3.0. jar.

jlgutenson commented 11 months ago

You could test by removing the GDAL 3.2. jar from the Vortex distribution and replacing with a GDAL 3.0. jar.

Bingo @tombrauer! That appears to be the issue. I replaced the gdal-3.2.0.jar in /lib/ with the the gdal-3.0.0.jar from here and Vortex ran correctly in Linux with this batch file.

VORTEX_HOME="/home/jlgutenson/vortex-0.11.0"
export PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal:$PATH"
export GDAL_DATA="$VORTEX_HOME/bin/gdal/gdal-data"
export PROJ_LIB="$VORTEX_HOME/bin/gdal/proj"
JAVA_VORTEX_PATH="$VORTEX_HOME/jre/bin/java"
JAVA_VORTEX_LIB_PATH="$VORTEX_HOME/bin:$VORTEX_HOME/bin/gdal"
CLASS_PATH="/home/jlgutenson/jython-standalone-2.7.3.jar:$VORTEX_HOME/lib/*"
export LD_LIBRARY_PATH="$VORTEX_HOME/bin/gdal"
$JAVA_VORTEX_PATH -Xmx12g -Djava.library.path="$JAVA_VORTEX_LIB_PATH" -classpath "$CLASS_PATH" org.python.util.jython met_data_import.py "/mnt/d/Gutenson_RATES/TWDB-FIF-LRGVDC/2023/Scripts/build_hms_inputs/mrms_pass2/2023082217" "/home/jlgutenson/model_library/HMS/RVD/gis/RVD_83_1/RVD_83_1.shp" "/home/jlgutenson/model_library/HMS/RVD/RVDJune2018_JLG_scripted_1.dss" MultiSensor_QPE_01H_Pass2_altitude_above_msl "MRMS"
tombrauer commented 11 months ago

I see, so the issue was that the GDAL jar version and natives version were out of sync. We'll have to make sure we update those to be synchronous. Thanks for helping us corner that one!

jlgutenson commented 11 months ago

Thank you guys for helping me corner this!

It's hard to run HMS with forecasts without being able to pre-process the forecast precipitation!

tombrauer commented 11 months ago

Should be fixed in this commit. This will go out in the next vortex release which should be v0.11.8.

jlgutenson commented 11 months ago

Thanks again @tombrauer! I'll go ahead and close this.