search

apache / hop

Hop Orchestration Platform
https://hop.apache.org/
Apache License 2.0
965 stars 349 forks source link

[Bug]: Helm deployment missing database drivers #4192

Closed alexray92 closed 2 months ago

alexray92 commented 2 months ago

Apache Hop version?

2.9.0 (helm list shows 2.5.0 but I see hop-core-2.9.0.jar in /opt/hop/lib/core)

Java version?

11.0.23

Operating system

Linux

What happened?

Edit: I believe this is a duplicate of https://github.com/apache/hop/issues/4013

Not sure if this is a bug or my not knowing what I'm doing, but I can't seem to get any database drivers to show up/work.

I installed Hop in microk8s using helm:

helm repo add apache-hop https://hop.apache.org
helm repo update
helm show values apache-hop/hop > values.yaml
k namespace create hop
# created PVC via rook-ceph
nano values.yml #set persistence.enabled=true and set existingClaim to a PVC
k create secret generic -n hop hop-deployment-server --from-literal=pass=XYZ123
helm install hop-deployment apache-hop/hop -f values.yaml -n hop

After adding an ingressroute via Traefik I got to Hop just fine. However, when adding a relational database connection, every "Connection Type" says "no driver installed". For example I tried MSSQL both normal and native and got these errors:

MS SQL Server (Native): driver class 'net.sourceforge.jtds.jdbc.driver' could not be found MS SQL Server: driver class 'net.sourceforge.jtds.jdbc.driver' could not be found

But there seemed to be MSSQL drivers for jdbc installed: Edit: I believe I should have looked at the 'web' pod in which case the issue is that the docker image is missing the jdbc directory per existing issue https://github.com/apache/hop/issues/4013

hop-deployment-server-6cc467cb58-tnhzc:/opt/hop$ find . -name "mssql"

./lib/jdbc/mssql-jdbc-12.4.1.jre11.jar
./plugins/actions/mssqlbulkload
./plugins/actions/mssqlbulkload/hop-action-mssqlbulkload-2.9.0.jar
./plugins/databases/mssql
./plugins/databases/mssql/hop-databases-mssql-2.9.0.jar
./plugins/databases/mssqlnative
./plugins/databases/mssqlnative/hop-databases-mssqlnative-2.9.0.jar

hop-deployment-server-56984bcd8d-t6c26:~$ ls /opt/hop/lib/jdbc

clickhouse-jdbc-0.6.0-patch4-all.jar  derbyshared-10.16.1.1.jar             jackson-annotations-2.15.0.jar        jna-platform-4.2.1.jar                org.osgi.enterprise-4.2.0.jar         sqlite-jdbc-3.43.0.0.jar
client-1.0.0-beta.2.jar               duckdb_jdbc-0.10.2.jar                jackson-core-2.15.0.jar               jt400-20.0.6.jar                      postgresql-42.7.2.jar                 waffle-jna-1.7.5.jar
common-1.0.0-beta.2.jar               guava-32.1.2-jre.jar                  jackson-databind-2.15.0.jar           monetdb-jdbc-3.3.jar                  redshift-jdbc42-2.1.0.28.jar
crate-jdbc-2.7.0.jar                  h2-2.2.224.jar                        jcl-over-slf4j-1.7.12.jar             mssql-jdbc-12.4.1.jre11.jar           slf4j-api-1.7.12.jar
derbyclient-10.16.1.1.jar             hsqldb-2.7.2.jar                      jna-4.2.1.jar                         org.osgi.core-4.3.1.jar               snowflake-jdbc-3.14.3.jar

hop-deployment-server-56984bcd8d-t6c26:~$ ls /opt/hop/plugins/databases/

access          cockroachdb     doris           generic         hive            ingres          mariadb         mysql           postgresql      sqlbase         teradata
as400           cratedb         duckdb          googlebigquery  hypersonic      interbase       monetdb         netezza         redshift        sqlite          universe
cache           db2             exasol4         greenplum       infobright      iris            mssql           oracle          sapdb           sybase          vectorwise
clickhouse      derby           firebird        h2              informix        kingbasees      mssqlnative     oraclerdb       snowflake       sybaseiq        vertica

I was guessing there should be an ENV variable by default in the values.yaml file, I tried defining server.image.env with HOP_SHARED_JDBC_FOLDERS set to either "/opt/hop/lib/jdbc" or "/opt/hop/lib/jdbc,/opt/hop/plugins/databases" and then running helm upgrade but that didn't seem to help despite it showing up on the running process:

hop-deployment-server-56984bcd8d-t6c26:~$ ps aux

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
hop            1  0.0  0.0   2356  1780 ?        Ss   21:37   0:00 /bin/bash /opt/hop/run.sh
hop            7  0.0  0.0   2448  1876 ?        S    21:37   0:00 /bin/bash /opt/hop/load-and-execute.sh
hop           12  0.0  0.0   2348  1748 ?        S    21:37   0:00 bash /opt/hop/hop-server.sh --level=Basic /tmp/hop-server.xml
hop           13  0.0  0.0   1616     4 ?        S    21:37   0:00 tee /opt/hop/hop.err.log
hop           37  8.4  5.9 10141556 977372 ?     Sl   21:37   0:12 java -XX:+AggressiveHeap -DHOP_AUDIT_FOLDER=./audit **-DHOP_SHARED_JDBC_FOLDERS=/opt/hop/lib/jdbc** -DHOP_PLATFORM_RUNTIME=Server -DHOP_AUTO_CREATE_CONFIG=Y -DHOP_PLATFORM_OS=Linux -Djava.library.path= -classpath lib/core/*:lib/beam/*:lib/swt/linux/x86_64/* org.apache.hop.www.HopServer --level=Basic /tmp/hop-server.xml
hop           70  0.0  0.0   2640  2308 pts/0    Ss   21:40   0:00 /bin/bash
hop           77  0.0  0.0   2524  1736 pts/0    R+   21:40   0:00 ps aux

Issue Priority

Priority: 2

Issue Component

Component: Containers, Component: Database, Component: Documentation, Component: Hop Config, Component: Hop Server

bamaer commented 2 months ago

this is fixed in the development version.