ImportError with snowflake.connector importing latest version pycryptodome

[swollentongue]

Python Version: 2.7 Snowflake connector version: 1.6.3

Recently we've been getting this error when running snowflake connector from freshly built docker image:

   File "/usr/local/lib/python2.7/site-packages/snowflake/connector/__init__.py", line 21, in <module>
     from .connection import SnowflakeConnection
   File "/usr/local/lib/python2.7/site-packages/snowflake/connector/connection.py", line 19, in <module>
     from .cursor import SnowflakeCursor
   File "/usr/local/lib/python2.7/site-packages/snowflake/connector/cursor.py", line 29, in <module>
     from .file_transfer_agent import (SnowflakeFileTransferAgent)
   File "/usr/local/lib/python2.7/site-packages/snowflake/connector/file_transfer_agent.py", line 31, in <module>
     from .s3_util import (SnowflakeS3FileEncryptionMaterial, SnowflakeS3Util,
   File "/usr/local/lib/python2.7/site-packages/snowflake/connector/s3_util.py", line 25, in <module>
     from Crypto.Cipher import AES
   File "/usr/local/lib/python2.7/site-packages/Crypto/Cipher/__init__.py", line 36, in <module>
     from Crypto.Cipher._mode_gcm import _create_gcm_cipher
   File "/usr/local/lib/python2.7/site-packages/Crypto/Cipher/_mode_gcm.py", line 47, in <module>
     from Crypto.Util import _cpuid
 ImportError: dynamic module does not define init function (init_cpuid)

This seems to be caused by a changes to pycryptodome as pinning the installed version to 3.6.2 fixes the issue.

[smtakeda]

Similar to https://github.com/Legrandin/pycryptodome/issues/178 But I'm not able to reproduce it on Mac Sierra 10.12.6. Can you provide the pip list output?

>pip list
Package                    Version
-------------------------- ---------
asn1crypto                 0.24.0
azure-common               1.1.13
azure-nspkg                2.0.0
azure-storage              0.36.0
boto3                      1.7.50
botocore                   1.10.50
certifi                    2018.4.16
cffi                       1.11.5
chardet                    3.0.4
cryptography               2.2.2
docutils                   0.14
enum34                     1.1.6
future                     0.16.0
futures                    3.2.0
idna                       2.7
ijson                      2.3
ipaddress                  1.0.22
jmespath                   0.9.3
pip                        10.0.1
pyasn1                     0.4.3
pyasn1-modules             0.2.2
pycparser                  2.18
pycryptodome               3.6.3
PyJWT                      1.6.4
pyOpenSSL                  17.5.0
python-dateutil            2.7.3
pytz                       2018.5
requests                   2.19.1
s3transfer                 0.1.13
setuptools                 28.8.0
six                        1.11.0
snowflake-connector-python 1.6.3
urllib3                    1.23
wheel                      0.29.0

And any file _cpuid* under /usr/local/lib/python2.7/site-packages/? My files are as follows.

>find /tmp/k/lib/python2.7/site-packages/ -name "_cpuid*" -print
/tmp/k/lib/python2.7/site-packages//Crypto/Util/_cpuid.py
/tmp/k/lib/python2.7/site-packages//Crypto/Util/_cpuid.pyc
/tmp/k/lib/python2.7/site-packages//Crypto/Util/_cpuid_c.so

[swollentongue]

My docker base image is ubuntu bionic beaver

Import this errors on: import snowflake.connector

output from pip list:

Package                    Version
-------------------------- ---------
asn1crypto                 0.24.0
azure-common               1.1.13
azure-nspkg                2.0.0
azure-storage              0.36.0
bingads                    11.5.4
bitarray                   0.8.2
boto3                      1.7.50
botocore                   1.6.8
certifi                    2018.4.16
cffi                       1.11.5
chardet                    3.0.4
Cheetah                    2.4.4
cryptography               2.2.2
docutils                   0.14
enum34                     1.1.6
future                     0.16.0
futures                    3.0.3
httplib2                   0.11.3
idna                       2.7
ijson                      2.3
ipaddress                  1.0.22
jmespath                   0.9.3
Markdown                   2.6.11
mysql-connector-python-rf  2.1.3
oauth2client               1.4.11
pip                        10.0.1
poster                     0.8.1
pyasn1                     0.2.3
pyasn1-modules             0.0.9
pybloom                    1.1
pycparser                  2.18
pycryptodome               3.6.3
PyJWT                      1.6.4
pyOpenSSL                  17.2.0
PySocks                    1.6.8
python-dateutil            2.7.3
python-gflags              2.0
pytz                       2018.5
PyYAML                     3.12
requests                   2.7.0
rsa                        3.1.4
s3transfer                 0.1.13
setuptools                 39.1.0
simplejson                 3.6.5
six                        1.11.0
snowflake-connector-python 1.6.3
suds-jurko                 0.6
virtualenv                 15.0.2
wheel                      0.29.0

find for cpuid:

root@e6e3aa55b239:/raptor# find /usr/local/lib/python2.7/site-packages/ -iname "_cpuid*"
/usr/local/lib/python2.7/site-packages/Crypto/Util/_cpuid.so
/usr/local/lib/python2.7/site-packages/Crypto/Util/_cpuid.py
/usr/local/lib/python2.7/site-packages/Crypto/Util/_cpuid_c.so
/usr/local/lib/python2.7/site-packages/Crypto/Util/_cpuid.pyc

[smtakeda]

Created a container based on Ubuntu 18.04 and tried but could not reproduce it either:

root@3ad3dda60789:/usr/bin# more /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04 LTS"
root@3ad3dda60789:/usr/bin# find /usr/ -name "_cpu*" -print
/usr/local/lib/python2.7/dist-packages/Crypto/Util/_cpuid.pyc
/usr/local/lib/python2.7/dist-packages/Crypto/Util/_cpuid_c.so
/usr/local/lib/python2.7/dist-packages/Crypto/Util/_cpuid.py

Here is the history:

apt update
apt upgrade
apt install -y python
apt install virtualenv
apt install python-pip
apt install vim
apt install curl
curl https://bootstrap.pypa.io/get-pip.py | python # upgraded pip to 10

pip output:

root@3ad3dda60789:/usr/bin# pip list
Package                    Version
-------------------------- ---------
asn1crypto                 0.24.0
azure-common               1.1.13
azure-nspkg                2.0.0
azure-storage              0.36.0
boto3                      1.7.50
botocore                   1.10.50
certifi                    2018.4.16
cffi                       1.11.5
chardet                    3.0.4
cryptography               2.1.4
docutils                   0.14
enum34                     1.1.6
future                     0.16.0
futures                    3.2.0
idna                       2.6
ijson                      2.3
ipaddress                  1.0.17
jmespath                   0.9.3
keyring                    10.6.0
keyrings.alt               3.0
pip                        10.0.1
pyasn1                     0.4.3
pyasn1-modules             0.2.2
pycparser                  2.18
pycrypto                   2.6.1
pycryptodome               3.6.3
pygobject                  3.26.1
PyJWT                      1.6.4
pyOpenSSL                  17.5.0
python-dateutil            2.7.3
pytz                       2018.5
pyxdg                      0.25
requests                   2.19.1
s3transfer                 0.1.13
SecretStorage              2.3.1
setuptools                 39.0.1
six                        1.11.0
snowflake-connector-python 1.6.3
urllib3                    1.23
wheel                      0.30.0

So as you have _cpuid.so, which is not included in 3.6.3, I suspect your base image already had older version of pycryptodome or even pycrypto? Here is the diff between last two versions:

>unzip -l pycryptodome-3.6.1-cp27-cp27mu-manylinux1_x86_64.whl | grep cpu
     8824  04-15-2018 20:04   Crypto/Util/_cpuid.so
>unzip -l pycryptodome-3.6.3-cp27-cp27mu-manylinux1_x86_64.whl | grep cpu                                                                                                                                      
    10931  06-21-2018 16:52   Crypto/Util/_cpuid_c.so
     1989  06-21-2018 16:49   Crypto/Util/_cpuid.py

[swollentongue]

That may entirely be the case since we are leveraging multiple docker build stages. I will try troubleshooting later today. Thanks for your help with troubleshooting this issue!