Closed davlee1972 closed 4 months ago
hi and thanks for submitting this - will check.
thanks for the pointers in this new issue ! so in #1033 , using the keypair.go
test program (which has sf.DSN
to generate the DSN) the issue was not reproducible.
initial investigations with a simple test program shows that when only using vanilla gosnowflake, both ParseDSN
and gosnowflake.Config
results in privatelink properly stripped out of the account name.
repro:
# cat my.go
package main
import (
"flag"
"fmt"
"os"
sf "github.com/snowflakedb/gosnowflake"
)
func main() {
if !flag.Parsed() {
flag.Parse()
}
accountName := os.Getenv("SFACCOUNT")
connectionString := "admin@" + accountName + ".privatelink/mydb/myschema?warehouse=compute_wh&authenticator=SNOWFLAKE_JWT"
config, err := sf.ParseDSN(connectionString)
if err != nil {
fmt.Println(err)
}
fmt.Println("===> Config from ParseDSN:")
fmt.Printf("%+v\n", config)
dsn, _ := sf.DSN(config)
fmt.Println("===> DSN from config generated by ParseDSN:")
fmt.Printf("%+v\n", dsn)
config = &sf.Config{
Account: accountName,
User: "admin",
Database: "mydb",
Schema: "myschema",
Warehouse: "compute_wh",
Authenticator: sf.AuthTypeJwt,
}
dsn, _ = sf.DSN(config)
fmt.Println("===> Config from manual Config:")
fmt.Printf("%+v\n", config)
fmt.Println("===> DSN from config defined manually:")
fmt.Printf("%+v\n", dsn)
}
MYACCOUNT
as accountname# export SFACCOUNT="MYACCOUNT" && go run my.go
===> Config from ParseDSN:
&{Account:MYACCOUNT User:admin Password: Database:mydb Schema:myschema Warehouse:compute_wh Role: Region:privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYACCOUNT.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:<nil> Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
admin:@MYACCOUNT.privatelink.snowflakecomputing.com:443?account=MYACCOUNT&authenticator=snowflake_jwt&database=mydb&ocspFailOpen=true®ion=privatelink&schema=myschema&validateDefaultParameters=true&warehouse=compute_wh
===> Config from manual Config:
&{Account:MYACCOUNT User:admin Password: Database:mydb Schema:myschema Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYACCOUNT.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:<nil> Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
admin:@MYACCOUNT.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=mydb&ocspFailOpen=true&schema=myschema&validateDefaultParameters=true&warehouse=compute_wh
MYORG-MYACCOUNT
as accountname# export SFACCOUNT="MYORG-MYACCOUNT" && go run my.go
===> Config from ParseDSN:
&{Account:MYORG-MYACCOUNT User:admin Password: Database:mydb Schema:myschema Warehouse:compute_wh Role: Region:privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYORG-MYACCOUNT.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:<nil> Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
admin:@MYORG-MYACCOUNT.privatelink.snowflakecomputing.com:443?account=MYORG-MYACCOUNT&authenticator=snowflake_jwt&database=mydb&ocspFailOpen=true®ion=privatelink&schema=myschema&validateDefaultParameters=true&warehouse=compute_wh
===> Config from manual Config:
&{Account:MYORG-MYACCOUNT User:admin Password: Database:mydb Schema:myschema Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYORG-MYACCOUNT.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:<nil> Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
admin:@MYORG-MYACCOUNT.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=mydb&ocspFailOpen=true&schema=myschema&validateDefaultParameters=true&warehouse=compute_wh
Which (the account name) is the same, as it is coming out of SnowSQL's --generate-jwt
function JWT in the iss
and sub
claims.
Do you have a way to reproduce this issue outside of ADBC ? Should reproduce easily, if one of the methods inside gosnowflake are faulty but did not see the error yet. Thank you in advance for the repro !
I think your test code with my.go should not add ".privatelink" to the account. Not every account has a .privatelink correct?
connectionString := "admin@" + accountName + ".privatelink/mydb/myschema?warehouse=compute_wh&authenticator=SNOWFLAKE_JWT"
Later there is.. Account: accountName,
I think the issue here is that .privatelink aka the region needs to stripped from account if there is one..
Here's what I'm seeing from ADBC with debugging on..
For adbc.snowflake.sql.account = "MYACCOUNT"
https://MYACCOUNT.snowflakecomputing.com:443/session/v1/login-request? etc.. context deadline exceeded (Client.Timeout exceeded while awaiting headers) This uri is obviously not valid..
For adbc.snowflake.sql.account = "MYACCOUNT.privatelink"
https://MYACCOUNT.privatelink.snowflakecomputing.com:443/session/v1/login-request? etc. IO: 390144 (08004): JWT token is invalid. [ca7259eb-5322-4e5c-bf5c-e58c48af08fa] The URL is correct now, but the JWT token is being generated with MYACCOUNT.privatelink when it should just be MYACCOUNT..
BTW using MYACCOUNT.privatelink with alternative authentication other than JWT works fine with the Go Driver..
I think I found the bug..
in DSN() the cfg.Region is captured before it is stripped out of the account..
// in case account includes region
posDot := strings.Index(cfg.Account, ".")
if posDot > 0 {
if cfg.Region != "" {
return "", errInvalidRegion()
}
cfg.Region = cfg.Account[posDot+1:]
cfg.Account = cfg.Account[:posDot]
But in parseDSN() I don't see any logic to check or capture the Region.
This impacts the final value of cfg.Host:
if cfg.Host == "" {
if cfg.Region != "" {
cfg.Host = cfg.Account + "." + cfg.Region + defaultDomain
} else {
cfg.Host = cfg.Account + defaultDomain
}
}
With no region and no host with a starting value of cfg.Account = MYACCOUNT.privatelink.. cfg.Account becomes cfg.Account = MYACCOUNT when .privatelink is stripped, but cfg.Host ends up with MYACCOUNT.snowflakecomputing.com which isn't valid since it is missing the region..
With no region and no host with a starting value of cfg.Account = MYACCOUNT cfg.Host ends up with MYACCOUNT.snowflakecomputing.com which isn't valid either..
hardcoded .privatelink
for the sake of reproduction of your case, but you're right, don't necessarily need to hardcode it. so new reproduction which isn't just displaying the parsed values, but also goes in and actually connects to snowflake.
my demo account is in AWS US WEST Oregon, so account format is
this is a live account which conforms to a regular customer account, all values were read from running SYSTEM$GET_PRIVATELINK_CONFIG()
in Snowflake as part of the privatelink setup.
using latest gosnowflake from main
.
Then to rule out any issues possibly related to us-west-2 (which is the default deployment) I repeated the test with an account in AWS EU CENTRAL Frankfurt.
repro, with actually connecting to Snowflake using JWT:
package main
import (
"flag"
"fmt"
"os"
"encoding/pem"
"crypto/rsa"
"crypto/x509"
"errors"
"database/sql"
"log"
sf "github.com/snowflakedb/gosnowflake"
)
func parsePrivateKeyFromFile(path string) (*rsa.PrivateKey, error) {
bytes, err := os.ReadFile(path)
if err != nil {
return nil, err
}
block, _ := pem.Decode(bytes)
if block == nil {
return nil, errors.New("failed to parse PEM block containing the private key")
}
privateKey, err := x509.ParsePKCS8PrivateKey(block.Bytes)
if err != nil {
return nil, err
}
pk, ok := privateKey.(*rsa.PrivateKey)
if !ok {
return nil, fmt.Errorf("interface convertion. expected type *rsa.PrivateKey, but got %T", privateKey)
}
return pk, nil
}
func doConnectAndQuery(dsn string, myquery string) {
db, err := sql.Open("snowflake", dsn)
if err != nil {
log.Fatalf("failed to connect. %v, err: %v", dsn, err)
}
defer db.Close()
query := myquery
rows, err := db.Query(query) // no cancel is allowed
if err != nil {
log.Fatalf("failed to run a query. %v, err: %v", query, err)
}
defer rows.Close()
var v string
for rows.Next() {
err := rows.Scan(&v)
if err != nil {
log.Fatalf("failed to get result. err: %v", err)
}
fmt.Println(v)
}
if rows.Err() != nil {
fmt.Printf("ERROR: %v\n", rows.Err())
return
}
}
func main() {
if !flag.Parsed() {
flag.Parse()
}
accountName := os.Getenv("SFACCOUNT")
userName := os.Getenv("SFUSER")
connectionString := userName + "@" + accountName + "/test_db/public?warehouse=compute_wh&authenticator=SNOWFLAKE_JWT"
config, err := sf.ParseDSN(connectionString)
pk, _ := parsePrivateKeyFromFile("rsa_key_unencrypted.p8")
config.PrivateKey = pk
if err != nil {
fmt.Println(err)
}
fmt.Println("===> Config from ParseDSN:")
fmt.Printf("%+v\n", config)
dsn, _ := sf.DSN(config)
fmt.Println("===> DSN from config generated by ParseDSN:")
fmt.Printf("%+v\n", dsn)
fmt.Println("===> Connecting to Snowflake - 1")
doConnectAndQuery(dsn, "SELECT 'ParseDSN with keypair';")
config = &sf.Config{
Account: accountName,
User: userName,
Database: "test_db",
Schema: "public",
Warehouse: "compute_wh",
Authenticator: sf.AuthTypeJwt,
PrivateKey: pk,
}
dsn, _ = sf.DSN(config)
fmt.Println("===> Config from manual Config:")
fmt.Printf("%+v\n", config)
fmt.Println("===> DSN from config defined manually:")
fmt.Printf("%+v\n", dsn)
fmt.Println("===> Connecting to Snowflake - 2")
doConnectAndQuery(dsn, "SELECT 'manual Config with keypair';")
}
$ for i in MYACCOUNT MYORG-MYACCOUNT MYACCOUNT.us-west-2.privatelink MYORG-MYACCOUNT.privatelink; do echo "===============> running for account name $i"; SFACCOUNT=$i go run my.go ; done
===============> running for account name MYACCOUNT
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYACCOUNT.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
dszmolka:@MYACCOUNT.snowflakecomputing.com:443?account=MYACCOUNT&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYACCOUNT.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
dszmolka:@MYACCOUNT.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
===============> running for account name MYORG-MYACCOUNT
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:MYORG-MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYORG-MYACCOUNT.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
dszmolka:@MYORG-MYACCOUNT.snowflakecomputing.com:443?account=MYORG-MYACCOUNT&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:MYORG-MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYORG-MYACCOUNT.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
dszmolka:@MYORG-MYACCOUNT.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
===============> running for account name MYACCOUNT.us-west-2.privatelink
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:us-west-2.privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYACCOUNT.us-west-2.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
dszmolka:@MYACCOUNT.us-west-2.privatelink.snowflakecomputing.com:443?account=MYACCOUNT&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=us-west-2.privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:us-west-2.privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYACCOUNT.us-west-2.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
dszmolka:@MYACCOUNT.us-west-2.privatelink.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=us-west-2.privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
===============> running for account name MYORG-MYACCOUNT.privatelink
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:MYORG-MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYORG-MYACCOUNT.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000320080 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
dszmolka:@MYORG-MYACCOUNT.privatelink.snowflakecomputing.com:443?account=MYORG-MYACCOUNT&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:MYORG-MYACCOUNT User:dszmolka Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:MYORG-MYACCOUNT.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000320080 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
dszmolka:@MYORG-MYACCOUNT.privatelink.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
$ for j in myotheraccount.eu-central-1 myotherorg-myotheraccount myotheraccount.eu-central-1.privatelink myotherorg-myotheraccount.privatelink; do echo "===============> running for account name $j"; SFACCOUNT=$j go run my.go ; done
===============> running for account name myotheraccount.eu-central-1
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:eu-central-1 ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotheraccount.eu-central-1.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000255080 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
admin:@myotheraccount.eu-central-1.snowflakecomputing.com:443?account=myotheraccount&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=eu-central-1&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:eu-central-1 ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotheraccount.eu-central-1.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000255080 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
admin:@myotheraccount.eu-central-1.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=eu-central-1&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
===============> running for account name myotherorg-myotheraccount
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:myotherorg-myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotherorg-myotheraccount.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
admin:@myotherorg-myotheraccount.snowflakecomputing.com:443?account=myotherorg-myotheraccount&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:myotherorg-myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region: ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotherorg-myotheraccount.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
admin:@myotherorg-myotheraccount.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
===============> running for account name myotheraccount.eu-central-1.privatelink
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:eu-central-1.privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotheraccount.eu-central-1.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
admin:@myotheraccount.eu-central-1.privatelink.snowflakecomputing.com:443?account=myotheraccount&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=eu-central-1.privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:eu-central-1.privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotheraccount.eu-central-1.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000254180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
admin:@myotheraccount.eu-central-1.privatelink.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=eu-central-1.privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
===============> running for account name myotherorg-myotheraccount.privatelink
WARN[0000]log.go:244 gosnowflake.(*defaultLogger).Warn DBUS_SESSION_BUS_ADDRESS envvar looks to be not set, this can lead to runaway dbus-daemon processes. To avoid this, set envvar DBUS_SESSION_BUS_ADDRESS=$XDG_RUNTIME_DIR/bus (if it exists) or DBUS_SESSION_BUS_ADDRESS=/dev/null.
===> Config from ParseDSN:
&{Account:myotherorg-myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotherorg-myotheraccount.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000234180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config generated by ParseDSN:
admin:@myotherorg-myotheraccount.privatelink.snowflakecomputing.com:443?account=myotherorg-myotheraccount&authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 1
ParseDSN with keypair
===> Config from manual Config:
&{Account:myotherorg-myotheraccount User:admin Password: Database:test_db Schema:public Warehouse:compute_wh Role: Region:privatelink ValidateDefaultParameters:1 Params:map[] ClientIP:<nil> Protocol:https Host:myotherorg-myotheraccount.privatelink.snowflakecomputing.com Port:443 Authenticator:SNOWFLAKE_JWT Passcode: PasscodeInPassword:false OktaURL:<nil> LoginTimeout:5m0s RequestTimeout:0s JWTExpireTimeout:1m0s ClientTimeout:15m0s JWTClientTimeout:10s ExternalBrowserTimeout:2m0s MaxRetryCount:7 Application:Go InsecureMode:false OCSPFailOpen:1 Token: TokenAccessor:<nil> KeepSessionAlive:false PrivateKey:0xc000234180 Transporter:<nil> DisableTelemetry:false Tracing: TmpDirPath: MfaToken: IDToken: ClientRequestMfaToken:0 ClientStoreTemporaryCredential:0 DisableQueryContextCache:false IncludeRetryReason:1 ClientConfigFile: DisableConsoleLogin:0}
===> DSN from config defined manually:
admin:@myotherorg-myotheraccount.privatelink.snowflakecomputing.com:443?authenticator=snowflake_jwt&database=test_db&ocspFailOpen=true&privateKey=<thisismykey>®ion=privatelink&schema=public&validateDefaultParameters=true&warehouse=compute_wh
===> Connecting to Snowflake - 2
manual Config with keypair
all 2 x 4 attempts (locator, regionless, privatelink locator, privatelink regionless) are successful when using only gosnowflake without ADBC.
So for now, I still did not find any bug in gosnowflake which would result in incorrect parse of region/privatelink. Account
is always correctly split, and goes correctly into the JWT.
Please observe I'm not using a manually specified Region
anywhere in the test, neither with ParseDSN
nor with gosnowflake.Config
attempts.
Yet, the Region
fields are correctly populated with ParseDSN
with gosnowflake.
Would it be possible for you to also try this simple test program above with your actual account, and see if you're hitting any errors without ADBC, and let me know how it went ? Please make sure a proper unencrypted private key exists at the specified location. Also might need to edit the DB name if 'test_db' doesn't exist.
is there any way to see the actual full url/connection string, which ADBC feeds into ParseDSN
? I'm using
myuser:mypassword@my_organization-my_account/mydb/testschema?warehouse=mywh
but this is not the only format which is accepted and it would be great to test ParseDSN
with the exact same format which comes from the upper application.
BTW using MYACCOUNT.privatelink with alternative authentication other than JWT works fine with the Go Driver..
Makes sense, because there is no JWT which is then wrongly populated by feeding incorrect configuration into gosnowflake, coming from ADF GUI or ADBC (or, if there's a bug with certain type of url/connection string, this is yet to be proved)
But in parseDSN() I don't see any logic to check or capture the Region.
I believe it comes from parseAccountHostPort
Ok I looked at the ADBC code and it is calling..
connector := gosnowflake.NewConnector(drv, *d.cfg)
The test code above is setting up a config and running DSN on it and then using the DSN to connect..
dsn, _ = sf.DSN(config)
There is logic in DSN() to strip Account into cfg.Account and cfg.Region.
// in case account includes region
posDot := strings.Index(cfg.Account, ".")
if posDot > 0 {
if cfg.Region != "" {
return "", errInvalidRegion()
}
cfg.Region = cfg.Account[posDot+1:]
cfg.Account = cfg.Account[:posDot]
}
err = fillMissingConfigParameters(cfg)
The Connect() function in Go Snowflake doesn't call DSN() or parseDSN().. It jumps directly to fillMissingConfigParameters().
// NewConnector creates a new connector with the given SnowflakeDriver and Config.
func NewConnector(driver InternalSnowflakeDriver, config Config) Connector {
return Connector{driver, config}
}
// Connect creates a new connection.
func (t Connector) Connect(ctx context.Context) (driver.Conn, error) {
cfg := t.cfg
err := fillMissingConfigParameters(&cfg)
if err != nil {
return nil, err
}
return t.driver.OpenWithConfig(ctx, cfg)
}
I think the right solution is to remove all the posDot functionality out of DSN() and ParseDSN() and just put it in fillMissingConfigParameters()..
fillMissingConfigParameters is already handling other quirks like "-" in cfg.Account..
func fillMissingConfigParameters(cfg *Config) error {
posDash := strings.LastIndex(cfg.Account, "-")
if posDash > 0 {
if strings.Contains(cfg.Host, ".global.") {
cfg.Account = cfg.Account[:posDash]
}
}
Here is the python snowflake connector code that handles ".", "-" and "global".. This logic isn't the same which is creating additional confusion how the same "account" used in the python driver / snowsql won't work in the Go driver..
def parse_account(account):
url_parts = account.split(".")
# if this condition is true, then we have some extra
# stuff in the account field.
if len(url_parts) > 1:
if url_parts[1] == "global":
# remove external ID from account
parsed_account = url_parts[0][0 : url_parts[0].rfind("-")]
else:
# remove region subdomain
parsed_account = url_parts[0]
else:
parsed_account = account
return parsed_account
Without logic to parse myaccount out of myaccount.privatelink in fillMissingConfigParameters the account value used to generate a JWT token is not valid..
The Azure Data Factory webui doesn't have the option to specify REGION or HOST independently from ACCOUNT so account cannot be treated as a pure account value in config.
Did you try running the above test program with your account name populated, without ADF ? I believe it should be successfuly able to connect with keypair, even when account name is regioned or has privatelink.
With that said, I'm trying to set up a repro to see how exactly ADF is sending the account/connection string and yes, probably the next step would be
reproduced the issue from ADF GUI and reading their specs I also did not find any ways to force Region
and to prevent ADF from generating an invalid JWT by sending an invalid Account
.
Even when the linked service was specified as raw JSON, the manually entered region
key was ignored thus the connection did not go to the correct account.
Whoever hits this same issue and not using privatelink; a quick workaround would be to use the regionless notation (myorg-myaccount
) as the account identifier in keypair authentication. Unfortunately with privatelink, this is not viable as for the connection to actually go through the private endpoint, the added .privatelink
suffix is necessary in the account name.
Of course a fully mitigating workaround, which works even with privatelink, to use Basic type of auth on the ADF GUI with the (sufficiently long and complex) password. This avoids JWT creation entirely.
We'll see how we can move forward with enhancing gosnowflake, but the issue itself needs to be optimally fixed on ADF GUI, perhaps by allowing users to specify region or parsing the account name correctly when Keypair auth is used.
I don’t use ADF and I’m not 100% sure why this issue was logged in ADBC, but I had initial problems with ADBC connections until I separated account and region when populating a config just for JWT authentication.
With basic and web authentication account.privatelink worked fine for the account parameter with the go driver in ADBC so it was confusing why changing the authentication method to JWT ended up in a failed connection.
It would be better to make the GO driver more resilient like the Python driver vs having every client like ADBC or ADF, etc implement REGION, etc..
on the gosnowflake side, prepared a change at https://github.com/snowflakedb/gosnowflake/pull/1146 which has been merged now and will be part of the next gosnowflake release cycle which should happen very soon as far as i know.
when creating the JWT token in gosnowflake, the new logic now checks what has been fed into it as the config.Account
and if something other than an actual Account
(e.g. locator account name or regioned account name or regionless with privatelink), it strips the Account
from it and proceeds to create JWT with only the Account
part.
How this version of gosnowflake with the extra Account
parsing logic will get into ADF (for those who are coming from ADF), I'm not really sure.
released in May 2024 cycle with version 1.10.1
Can we reopen this one? https://github.com/snowflakedb/gosnowflake/issues/1033
The test was using keypair.go to test JWT authentication and keypair.go calls DSN() under the hood.
The ADBC Go Snowlake driver is calling parseDSN() or is populating gosnowflake.Config to create a DSN. Both methods are missing logic to strip region from the account so ".privatelink" ends up in the JWT token id which makes it invalid..
https://github.com/apache/arrow-adbc/issues/1777 https://github.com/apache/arrow-adbc/issues/1422