SNOW-802933: Can't customise cache dir for OCSP response validation caches so an in memory cache is used instead

[lyderichti59]

1. What version of Python are you using? => Python 3.10.9
2. What operating system and processor architecture are you using? => Linux-5.4.0-1100-aws-x86_64-with-glibc2.27
3. What are the component versions in the environment (pip freeze)?

asn1crypto==1.5.1
build==0.10.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
cryptography==40.0.2
filelock==3.12.0
idna==3.4
oscrypto==1.3.0
packaging==23.1
pycparser==2.21
pycryptodomex==3.17
PyJWT==2.6.0
pyOpenSSL==23.1.1
pyproject_hooks==1.0.0
pytz==2023.3
requests==2.29.0
snowflake-connector-python==3.0.3
typing_extensions==4.5.0
urllib3==1.26.15

4. What did you do?

I ran the below python snippet using python-snowflake-connector, with the below environment variables, from an AWS lambda: Env vars (I don't think they are documented, but I found them in the source code) :

SF_TEMPORARY_CREDENTIAL_CACHE_DIR=/tmp/.cache/snowflake
SF_OCSP_RESPONSE_CACHE_DIR=/tmp/.cache/snowflake

Python snippet

   # NOTE : running below with OCSP mode set to : Fail Close
    with snowflake.connector.connect(**snowflake_connection_parameters) as connection: 
        cursor = connection.cursor(DictCursor)
        cursor.execute("SELECT 1 as N FROM dual", {})
        return cursor.fetchall()

5. What did you expect to see?

I was expecting to see no failure in the DEBUG logs and to have caches dumped to files. However, the DEBUG logs show that although I would like to have my cache under the /tmp folder, the connector still attempts to write some cache to my home folder. As I run on a AWS lambda, my /home/<user> file system is not writeable, it's a readonly system. The connector will then fallback to using in memory caches here, but I'd prefer writing to files as a benchmark I did makes the connection initialisation faster.

By looking at the code here : https://github.com/snowflakedb/snowflake-connector-python/blob/main/src/snowflake/connector/ocsp_snowflake.py#L85 The path where we want to write to, for the OCSP validation cache, is hardcoded and there's no way to customise this path.

Tested locally with a draft change to the source code : If we introduce an env var, such as the one used above, e.g. SF_OCSP_RESPONSE_VALIDATION_CACHE_DIR and set it to a writable path, and use it when building the file_path dict pointed to above, then the issue disappear and we use SFDictFileCaches as expected.

Being able to specify cache directories on lambda environments makes the lambdas execution faster.

6. Can you set logging to DEBUG and collect the logs? (Username and links obfuscated)

   2023-04-27 12:12:28,280 - MainThread connectionpool.py:1003 - _new_conn() - DEBUG - Starting new HTTPS connection (1): **************.snowflakecomputing.com:443
   2023-04-27 12:12:28,349 - MainThread ocsp_snowflake.py:92 - ocsp_response_validation_cache_dicts() - DEBUG - cache directory: /home/****/.cache/snowflake
   2023-04-27 12:12:28,350 - MainThread ocsp_snowflake.py:97 - ocsp_response_validation_cache_dicts() - DEBUG - cannot create a cache directory: [/home/*****/.cache/snowflake], err=[[Errno 13] Permission denied: '/home/*****/.cache/snowflake']
   2023-04-27 12:12:28,351 - MainThread network.py:1167 - _use_requests_session() - DEBUG - Session status for SessionPool '***********.snowflakecomputing.com', SessionPool 0/1 active sessions

[sfc-gh-mkeller]

I'm actually working on something that'll help this

[piyushbhojgude]

Has this issue been addressed ?