Handle 390114 "Authentication token has expired" response from server

[yemble]

Issue description

We've been receiving the following error from Snowflake:

390114: Authentication token has expired.  The user must authenticate again.

It's not being caught and handled in the driver. restful.go has

sessionExpiredCode       = "390112"

Perhaps the error number has changed, or this is a new variant which needs to be handled the same way?

Error log

As above

Configuration

Long running kafka->snowflake process.

[smtakeda]

There are two tokens issued for the session, master and session token. The master token is valid for 4 hours, while the session token is valid 1 hours. Obviously the session token expires first, 390112 is returned, then the driver will automatically renew it using the master token and update the master token itself. So in theory, if the token renewal occurs every hour, the connection is valid forever.

However in order to renew tokens, the clients require to communicate with the server continuously, so if the session is idle for 4 hours or more, both master and session tokens get expired, and there is no way to renew them. That's the timing when you get 390114.

390112 is used internally for token renew, so the application doesn't need to worry about it, but 390114 requires to be caught and ask the user and password to the users.

One way to mitigate this is that the application keeps running a query, for example select 1 every hour to force the tokens are refreshed.

JDBC and ODBC has heartbeat option but not implemented in Gosnowflake yet.

[yemble]

Thanks for that explanation. Reopening the connection will suffice until the heartbeat feature is implemented. In practice we're unlikely to be idle for such a long time.

[IkiM0no]

@smtakeda we are facing this identical 390114 issue for an app that requires a long-lived snowflake connection, but is likely to be called infrequently.

Running select 1 every hour feels to us like a hack, as does re-opening the connection feel non-idiomatic as db.SQL creates a pool of connections and manages them for you, but it never closes them.

We are at the point of considering patching gosnowflake to refresh the master token when 390114 is returned, but this seems less than ideal as we don't know the code base.

Is there an ETA on the heartbeat feature?

[smtakeda]

No ETA at the moment. I'll try finding my time (or hiring somebody) to get this done.

[jerry-b]

We had to work around this as well, the driver will be much more convenient when there is a built in solution.

[smtakeda]

An initial implementation. A simple test passed. https://github.com/snowflakedb/gosnowflake/pull/164

Would be great if somebody reviews the code.

[rem7]

@smtakeda any particular reason why you guys are choosing to do a heartbeat? I believe usually clients (mysql, postgres) that implement database/sql simply auto-reconnect if the connection gets broken. Why not just issue a reconnect in the background when 390114 is encountered?

[smtakeda]

@rem7 If the authentication method is a pair of user/passwd, the driver could keep it in memory for later use, but that's not what we want for security reason. For example, serialized data including secrets could be stolen. A thread associated with other account could read it from memory. In fact, most of Snowflake drivers wipe out the credentials right after the authentication to reduce a chance of the incidents. Also in SSO authentication, the token it gets earlier may not be valid at the time of reconnection, and no password is given to the driver, so there is no way to reconnect automatically without asking the password or popping up a browser and walking through SSO roundtrip.

[rem7]

yes, for user/pass that are services it should be stored in memory. A lot of applications that use drivers like this will be using service roles that aren't SSO. Most drivers store it in memory because they received this data passed to them, for example sql.Open("snowflake", dataSourceName).

The password came from a configuration file, file on disk, env var, etc. Needless to say, your library won't be able to control the security for dataSourceName, its outside of your security boundry.

Take a look at mysql for example, they store it on a Config struct: https://github.com/go-sql-driver/mysql/blob/9181e3a86a19bacd63e68d43ae8b7b36320d8092/dsn.go#L34

Or Postgres, just a simple map[string]string :) https://github.com/lib/pq/blob/a96442e255fce502751c604916b0e14e81be6bf9/conn.go#L109-L111

[smtakeda]

Again we won't cache secrets. That's application's choice and not default behavior of the driver. The session refresh will work for all authentication methods.

[williammarino]

@smtakeda Can we add this to the python driver as well? Getting the same error.

[bparhy]

@smtakeda I just had the same error while running this using python and aws lambda. Please let me know which version of driver has the fix for it for python?

[smtakeda]

@bparhy https://pypi.org/project/snowflake-connector-python/ 1.6.10+ has the support. I would recommend you to use the latest one 1.7.4.

[slin-aurascape]

I think this is a driver issue. Nothing related with authentication timeout. I have a the case the snowflake received a couple insert queries from the same client, and at the same minutes, the drive report 390114. See above log, 3 INSERT queries at 3:26 pm PST summer time from Snowflake backend log. Then from our client log, you can see we got authentication error 390114 at 22:26:34 UTC time. The same minute we had previous 3 queries.