search

snowflakedb / snowpark-java-scala

Snowflake Snowpark Java & Scala API
Apache License 2.0
18 stars 20 forks source link

SNOW-1038770: Failed to initialize MemoryUtil. Was Java started with `--add-opens=java.base/java.nio=ALL-UNNAMED`? #82

Open knanne opened 10 months ago

knanne commented 10 months ago

Trying to bump our our Springboot application to Java 17, was excited to see 17 support in new release of snowpark 1.10

However, we are still getting the below error message. The error happens immediately after performing an INSERT statement on the database successfully. The code doing that insert is 

...
df_log.write().option("columnOrder", "name").mode(SaveMode.Append).saveAsTable("AUDIT_JOB_LOGS");

session.close();

It looks like the release included passing of some java args https://github.com/snowflakedb/snowpark-java-scala/pull/72/commits/8b4c5796868ff4ca390f7c03db3f0b58ee4bfd2b as described as a "short-term solution" in https://community.snowflake.com/s/article/JDBC-Driver-Compatibility-Issue-With-JDK-16-and-Later

Is this maybe not excluding something important?

My build.gradle file is also below

error

[restartedMain] INFO com.snowflake.snowpark.Session - Closing stderr and redirecting to stdout
[restartedMain] INFO com.snowflake.snowpark.Session - Done closing stderr and redirecting to stdout
[restartedMain] INFO com.snowflake.snowpark.internal.ParameterUtils - set JDBC client memory limit to 10240
[restartedMain] INFO com.snowflake.snowpark.Session - Snowpark Session information: {
 "snowpark.version" : "1.10.0",
 "client.language": "Java",
 "java.version" : "17.0.9",
 "scala.version" : "2.12.18",
 "jdbc.session.id" : "66284275835526",
 "os.name" : "Windows 10",
 "jdbc.version" : "3.14.4",
 "snowpark.library" : "/C:/Users/kain.nanne/.gradle/caches/modules-2/files-2.1/com.snowflake/snowpark/1.10.0/c13dc54ec6cdd0e2468d7f3ec48ccc68ffbd2e7a/snowpark-1.10.0.jar",
 "scala.library" : "/C:/Users/kain.nanne/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-library/2.12.18/fa825f984172f6801e3d8c476071b1823c0a925b/scala-library-2.12.18.jar",
 "jdbc.library" : "/C:/Users/kain.nanne/.gradle/caches/modules-2/files-2.1/net.snowflake/snowflake-jdbc/3.14.4/aa39560c63fc8a578aecf6a0314d3cc7e0ae507/snowflake-jdbc-3.14.4.jar"
}
[restartedMain] INFO com.snowflake.snowpark.internal.ServerConnection - Actively querying parameter QUERY_TAG from server.
[restartedMain] INFO com.snowflake.snowpark.internal.ServerConnection - Execute query [queryID: 01b21079-0000-b063-0000-3c491284b042] DESCRIBE TABLE AUDIT_JOB_LOGS
[restartedMain] INFO com.snowflake.snowpark.internal.ServerConnection - Execute query [queryID: 01b21079-0000-b07d-0000-3c491284ad7e] DESCRIBE TABLE AUDIT_JOB_LOGS
[restartedMain] INFO com.snowflake.snowpark.internal.ServerConnection - Execute query [queryID: 01b21079-0000-b063-0000-3c491284b082]  INSERT  INTO AUDIT_JOB_LOGS SELECT  *  FROM ( SELECT "JOB_ID", "SERVER", "JOB_TYPE", "START_TIME", "END_TIME", "ARGS", "RECORDS", "DURATION_SEC", "STATUS", "ERROR", "COMMENT" FROM ( SELECT "JOB_ID", "SERVER", "JOB_TYPE", "START_TIME", to_timestamp("END_TIME") AS "END_TIME", "ARGS", "RECORDS", "DURATION_SEC", "STATUS", "ERROR", "COMMENT" FROM ( SELECT  *  FROM ( VALUES ('5bdcf6c1-59bd-4655-adad-540081387340', 'AVD0629328-0: kain.nanne', 'JAVA APP STARTUP', NULL :: string, '2024-02-01 12:01:10', NULL :: string, NULL :: int, NULL :: int, 'COMPLETE', NULL :: string, 'LOG: INFO | BATCH SIZE: 100000 | SERVER TIME: 2024-02-01 12:01:10 | NEXT SCHEDULE: 2024-02-02 06:00:00') AS SNOWPARK_TEMP_TABLE_ON89TG4BKZYLF13("JOB_ID", "SERVER", "JOB_TYPE", "START_TIME", "END_TIME", "ARGS", "RECORDS", "DURATION_SEC", "STATUS", "ERROR", "COMMENT")))))
[restartedMain] INFO com.snowflake.snowpark.Session - Closing session: {
 "snowpark.version" : "1.10.0",
 "client.language": "Java",
 "java.version" : "17.0.9",
 "scala.version" : "2.12.18",
 "jdbc.session.id" : "66284275835526",
 "os.name" : "Windows 10",
 "jdbc.version" : "3.14.4",
 "snowpark.library" : "/C:/Users/kain.nanne/.gradle/caches/modules-2/files-2.1/com.snowflake/snowpark/1.10.0/c13dc54ec6cdd0e2468d7f3ec48ccc68ffbd2e7a/snowpark-1.10.0.jar",
 "scala.library" : "/C:/Users/kain.nanne/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-library/2.12.18/fa825f984172f6801e3d8c476071b1823c0a925b/scala-library-2.12.18.jar",
 "jdbc.library" : "/C:/Users/kain.nanne/.gradle/caches/modules-2/files-2.1/net.snowflake/snowflake-jdbc/3.14.4/aa39560c63fc8a578aecf6a0314d3cc7e0ae507/snowflake-jdbc-3.14.4.jar"
}
[restartedMain] INFO com.snowflake.snowpark.Session - Canceling all running query
java.lang.RuntimeException: Failed to initialize MemoryUtil. Was Java started with `--add-opens=java.base/java.nio=ALL-UNNAMED`? (See https://arrow.apache.org/docs/java/install.html)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:146)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:231)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:226)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:728)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:67)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:145)
        at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.setFirstChunkRowCountForArrow(SnowflakeResultSetSerializableV1.java:1127)
        at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:601)
        at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:497)
        at net.snowflake.client.core.SFResultSetFactory.getResultSet(SFResultSetFactory.java:34)
        at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:234)
        at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:140)
        at net.snowflake.client.core.SFStatement.execute(SFStatement.java:776)
        at net.snowflake.client.core.SFStatement.execute(SFStatement.java:684)
        at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:267)
        at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:122)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$runQueryGetResult$1(ServerConnection.scala:388)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.runQueryGetResult(ServerConnection.scala:383)
        at com.snowflake.snowpark.internal.ServerConnection.runQuery(ServerConnection.scala:359)
        at com.snowflake.snowpark.Session.cancelAll(Session.scala:154)
        at com.snowflake.snowpark.Session.close(Session.scala:1243)
        at com.snowflake.snowpark_java.Session.close(Session.java:484)
        at fhirtransformer.engine.SnowflakeConfig.logJavaJobStarted(SnowflakeConfig.java:153)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:237)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:168)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:149)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384)
        at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109)
        at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
        at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:352)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1358)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1347)
        at fhirtransformer.StreamApp.main(StreamApp.java:12)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:50)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @4617c264
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:91)
        ... 51 more
[restartedMain] ERROR com.snowflake.snowpark.internal.ServerConnection - failed to execute query:
select system$cancel_all_queries(66284275835526)
[restartedMain] ERROR org.springframework.boot.SpringApplication - Application run failed
com.snowflake.snowpark.SnowparkClientException: Error Code: 0415, Error message: Failed to close this session. The error is: JDBC driver internal error: exception creating result java.lang.ExceptionInInitializerError at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:231).
        at com.snowflake.snowpark.internal.ErrorMessage$.createException(ErrorMessage.scala:420)
        at com.snowflake.snowpark.internal.ErrorMessage$.MISC_FAILED_CLOSE_SESSION(ErrorMessage.scala:379)
        at com.snowflake.snowpark.Session.close(Session.scala:1247)
        at com.snowflake.snowpark_java.Session.close(Session.java:484)
        at fhirtransformer.engine.SnowflakeConfig.logJavaJobStarted(SnowflakeConfig.java:153)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:237)
        at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:168)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:178)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:171)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:149)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384)
        at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109)
        at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
        at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:352)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1358)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1347)
        at fhirtransformer.StreamApp.main(StreamApp.java:12)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:50)

build.gradle

plugins {
    id 'java'
    id 'java-library'
    id 'org.springframework.boot' version '3.2.1'
    id 'org.sonarqube' version '4.2.1.3168'
    id 'jacoco'
    id 'io.freefair.lombok' version '8.4'
}

apply plugin: 'io.spring.dependency-management'

group = 'fhirtransformer'
version = '1.0'
description = 'matchboxtransformer'
sourceCompatibility = 17

repositories {
    mavenCentral()
}

dependencies {
    developmentOnly 'org.springframework.boot:spring-boot-devtools'
    implementation 'org.springframework.boot:spring-boot'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-autoconfigure'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.json:json:20230618'
    implementation 'org.javatuples:javatuples:1.2'
    testImplementation 'junit:junit:4.11'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.awaitility:awaitility:3.1.2'
    api 'com.snowflake:snowpark:1.10.0'
    api 'health.matchbox:matchbox-engine:3.3.3'
}

configurations.all {
   exclude group: 'ch.qos.logback', module: 'logback-core'
}

test {
    useJUnitPlatform()
}

jar {
    manifest {
        attributes 'Main-Class': 'fhirtransformer.StreamApp'
    }
}
knanne commented 10 months ago

Interesting enough, if I remove session.close(); the error is not thrown

However, when performing a SELECT later on, a similar error comes back.

Specifically this code block (where limitNum = 10)

log.info("Querying Snowflake");
DataFrame df_src = session.table(sourceTable).filter(Functions.col("TRANSFORMED_IND").equal_to(Functions.lit(0)));
if (limitNum>0) {
    log.info("Limiting Query to "+limitNum+" records");
    df_src = df_src.limit(limitNum);
}
long records_num = df_src.count();

fails with, error

[http-nio-8080-exec-2] INFO fhirtransformer.engine.Transformer - Querying Snowflake
[http-nio-8080-exec-2] INFO fhirtransformer.engine.Transformer - Limiting Query to 10 records
[2024-02-01 16:22:01.817] - 9660 SEVERE [http-nio-8080-exec-2] --- net.snowflake.client.core.SFStatement: Exception creating result
java.lang.NoClassDefFoundError: Could not initialize class net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:231)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:226)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:728)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:67)
        at net.snowflake.client.jdbc.internal.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:145)
        at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.setFirstChunkRowCountForArrow(SnowflakeResultSetSerializableV1.java:1127)
        at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:601)
        at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:497)
        at net.snowflake.client.core.SFResultSetFactory.getResultSet(SFResultSetFactory.java:34)
        at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:234)
        at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:140)
        at net.snowflake.client.core.SFStatement.execute(SFStatement.java:776)
        at net.snowflake.client.core.SFStatement.execute(SFStatement.java:684)
        at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:267)
        at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:122)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$runQueryGetResult$1(ServerConnection.scala:388)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.runQueryGetResult(ServerConnection.scala:383)
        at com.snowflake.snowpark.internal.analyzer.Query.runQueryGetResult(SnowflakePlan.scala:798)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$executePlanInternal$2(ServerConnection.scala:750)
        at com.snowflake.snowpark.internal.analyzer.SnowflakePlan$.wrapException(SnowflakePlan.scala:159)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$executePlanInternal$1(ServerConnection.scala:723)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.executePlanInternal(ServerConnection.scala:723)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$execute$1(ServerConnection.scala:689)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.execute(ServerConnection.scala:689)
        at com.snowflake.snowpark.DataFrame.collect(DataFrame.scala:2225)
        at com.snowflake.snowpark.DataFrame.count(DataFrame.scala:2253)
        at com.snowflake.snowpark_java.DataFrame.count(DataFrame.java:791)
        at fhirtransformer.engine.Transformer.transform(Transformer.java:1387)
        at fhirtransformer.APIController.TransformEndpoint(APIController.java:63)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:262)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:190)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:917)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:829)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.RuntimeException: Failed to initialize MemoryUtil. Was Java started with `--add-opens=java.base/java.nio=ALL-UNNAMED`? (See https://arrow.apache.org/docs/java/install.html) [in thread "http-nio-8080-exec-5"]
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:146)
        ... 87 more

[http-nio-8080-exec-2] ERROR com.snowflake.snowpark.internal.ServerConnection - failed to execute query:
 SELECT count(1) AS "COUNT(LITERAL())" FROM ( SELECT  *  FROM ( SELECT  *  FROM ( SELECT  *  FROM (SOURCE_RESOURCE)) WHERE ("TRANSFORMED_IND" = 0 :: int)) LIMIT 10) LIMIT 1
[2024-02-01 16:22:01.858] - 9660 SEVERE [http-nio-8080-exec-2] --- org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: net.snowflake.client.jdbc.SnowflakeSQLException: JDBC driver internal error: exception creating result java.lang.NoClassDefFoundError: Could not initialize class net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:231).] with root cause
net.snowflake.client.jdbc.SnowflakeSQLException: JDBC driver internal error: exception creating result java.lang.NoClassDefFoundError: Could not initialize class net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil at net.snowflake.client.jdbc.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:231).
        at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:276)
        at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:122)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$runQueryGetResult$1(ServerConnection.scala:388)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.runQueryGetResult(ServerConnection.scala:383)
        at com.snowflake.snowpark.internal.analyzer.Query.runQueryGetResult(SnowflakePlan.scala:798)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$executePlanInternal$2(ServerConnection.scala:750)
        at com.snowflake.snowpark.internal.analyzer.SnowflakePlan$.wrapException(SnowflakePlan.scala:159)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$executePlanInternal$1(ServerConnection.scala:723)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.executePlanInternal(ServerConnection.scala:723)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$execute$1(ServerConnection.scala:689)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:925)
        at com.snowflake.snowpark.internal.ServerConnection.execute(ServerConnection.scala:689)
        at com.snowflake.snowpark.DataFrame.collect(DataFrame.scala:2225)
        at com.snowflake.snowpark.DataFrame.count(DataFrame.scala:2253)
        at com.snowflake.snowpark_java.DataFrame.count(DataFrame.java:791)
        at fhirtransformer.engine.Transformer.transform(Transformer.java:1387)
        at fhirtransformer.APIController.TransformEndpoint(APIController.java:63)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:262)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:190)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:917)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:829)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:840)
sfc-gh-bli commented 9 months ago

It is a JDBC limitation. Snowflake JDBC replies dependencies such as Apache Arrow, which require those Java libs. https://arrow.apache.org/docs/java/install.html#id3

sfc-gh-jfreeberg commented 9 months ago

@knanne -- Sorry for the late reply. Can you set --add-opens=java.base/java.nio=ALL-UNNAMED as described in this doc: https://docs.snowflake.com/en/developer-guide/snowpark/java/troubleshooting#unnamed-module-error-on-java-17

As Bing mentioned, Apache Arrow requires this parameter.