Update Snowflake Drivers

[evantahler]

This announcement is to let you know that all partner and customer Snowflake accounts hosted on Google Cloud must update their client drivers to at least the minimum versions listed below by November 1, 2023. These new minimum versions, and all subsequent versions, provide critical fixes needed to continue accessing Snowflake on Google Cloud. Snowflake will therefore no longer support versions older than the new minimum versions for Google Cloud accounts.

Please note that the deprecation does not impact AWS or Azure partners or customers, who can continue to use driver versions released within the last two years as per the Snowflake Support policy.

WHY ARE CLIENT UPGRADES REQUIRED FOR GOOGLE CLOUD CUSTOMERS? Historically, presigned URL API requests to Snowflake GCP buckets only required the payload to be signed. However Google now requires both the request headers and payload to be signed. After the cutover date, once this requirement is enforced, GET and PUT requests to the stage from clients below the minimum version will fail with HTTP Error 400.

To avoid these failures, Snowflake has decided in consultation with Google to switch all drivers from using presigned URL authentication to using GCS downscoped token authentication.

ACTION TO TAKE: In response to Google’s changes, Snowflake has released a new set of drivers. All customers with their Snowflake account hosted on Google Cloud must update their client drivers to these minimum versions or later listed below by November 1, 2023.

Partners should also upgrade drivers for their SaaS environments, and should test, and bundle, or recommend at least the minimum supported driver versions for GCP accounts for their customer managed environments. Note that all customers with Snowflake accounts hosted on GCP will receive a communication about this change, so they may reach out to you to ensure that you’re using a supported client driver.

Snowflake will ensure that the older drivers continue to work for GCP accounts until Nov 1, 2023. They will be supported as per the Snowflake Client and Version Support policy for this duration.

After the cutover date, the GET and PUT requests from client drivers below the minimum version will fail with HTTP Error 400. Please see this KB article for more details. Please see this article for more information on how to check your current client driver versions in Snowflake.

[evantahler]

In https://github.com/airbytehq/airbyte/pull/25446/ I tried to see if we could just bump the version of the driver and everything would be fine... and it wasn't :(

[evantahler]

The bug might be with our test harness when are reading the data to confirm what we wrote

[evantahler]

Decision - Let's remove normalization + snowfalke and revisit

[evantahler]

This issue is back on the table due to potential bugs with file uploading. See https://github.com/airbytehq/oncall/issues/1493 and #bug-snowflake-driver for details

[evantahler]

From @ryankfu

Hypothesizing here that our retry logic may have a :bug: https://github.com/airbytehq/airbyte/blob/0f39af75b63d54beed9cfbeec2c12f7caa40d2e6[…]estination/snowflake/SnowflakeInternalStagingSqlOperations.java Another option that could be tested first is to set some queryTimeout parameter in the JDBC params for Snowflake (this is probably the fastest option although replication would be hard since what triggers this infinite retry isn't abundantly clear)

[edgao]

just for posterity, stacktrace of one of the failed test cases from https://github.com/airbytehq/airbyte/pull/25446

``` SnowflakeS3CopyDestinationAcceptanceTest > testIncrementalDedupeSync() FAILED java.lang.RuntimeException: net.snowflake.client.jdbc.SnowflakeLoggedFeatureNotSupportedException at io.airbyte.db.jdbc.JdbcDatabase$1.tryAdvance(JdbcDatabase.java:86) at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at io.airbyte.db.jdbc.DefaultJdbcDatabase.bufferedResultSetQuery(DefaultJdbcDatabase.java:56) at io.airbyte.integrations.destination.snowflake.SnowflakeInsertDestinationAcceptanceTest.retrieveRecordsFromTable(SnowflakeInsertDestinationAcceptanceTest.java:143) at io.airbyte.integrations.destination.snowflake.SnowflakeInsertDestinationAcceptanceTest.retrieveRecords(SnowflakeInsertDestinationAcceptanceTest.java:110) at io.airbyte.integrations.standardtest.destination.DestinationAcceptanceTest.retrieveRawRecordsAndAssertSameMessages(DestinationAcceptanceTest.java:1382) at io.airbyte.integrations.standardtest.destination.DestinationAcceptanceTest.testIncrementalDedupeSync(DestinationAcceptanceTest.java:823) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at jdk.proxy2/jdk.proxy2.$Proxy5.stop(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193) at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65) at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) Caused by: net.snowflake.client.jdbc.SnowflakeLoggedFeatureNotSupportedException at app//net.snowflake.client.jdbc.SnowflakeBaseResultSet.getObject(SnowflakeBaseResultSet.java:1317) at app//com.zaxxer.hikari.pool.HikariProxyResultSet.getObject(HikariProxyResultSet.java) at app//io.airbyte.db.jdbc.AbstractJdbcCompatibleSourceOperations.getObject(AbstractJdbcCompatibleSourceOperations.java:224) at app//io.airbyte.db.jdbc.AbstractJdbcCompatibleSourceOperations.putTimestampWithTimezone(AbstractJdbcCompatibleSourceOperations.java:234) at app//io.airbyte.db.jdbc.JdbcSourceOperations.copyToJsonField(JdbcSourceOperations.java:61) at app//io.airbyte.db.jdbc.AbstractJdbcCompatibleSourceOperations.rowToJson(AbstractJdbcCompatibleSourceOperations.java:55) at app//io.airbyte.db.jdbc.JdbcDatabase$1.tryAdvance(JdbcDatabase.java:83) ... 97 more ```

[evantahler]

Note: we are currently running a fork of the snowflake JDBC driver which has a fix they have yet to merge. We will likely need to do the same for the later versions. @edgao will move this fork to the airbyte repo shortly.

[edgao]

https://github.com/airbytehq/snowflake-jdbc now exists! for reference:

• https://github.com/edgao/snowflake-jdbc/pull/1 is how I built the 3.13.19 jar file
• https://github.com/snowflakedb/snowflake-jdbc/pull/1396 is the PR to get this change into the official repo

[evantahler]

Issue opened with snowflake: https://github.com/snowflakedb/snowflake-jdbc/issues/1431

Hey Evan, I have opened an issue in the snowflake-jdbc repository with my findings. I have tried debugging the jdbc driver but the error doesn't really make sense since nothing substantial was changed between 3.13.19 and 3.13.20 when the error starts occurring. I will wait for their input to see if this is an expected behaviour going forward and if we are going to be required to implement our own mappers and will act accordingly.

[evantahler]

@itaseskii looks like snowflake responeded!

[evantahler]

@itaseskii - it's been a minute! Any update?

[evantahler]

We need to get back to working on this. Un-assigning from @itaseskii as we haven't heard back from him, and moving back to our backlog.