search

dlenski / python-vipaccess

A free software implementation of Symantec's VIP Access application and protocol
Apache License 2.0
828 stars 84 forks source link

vipaccess fails on Mac OS Ventura run on Apple silicon .. pycryptodome load issue #68

Closed beaufort2015 closed 10 months ago

beaufort2015 commented 10 months ago

python-vipaccess installed with --user option

Invoking vipaccess provision fails with:

:/Users/noid
-> vipaccess provision -p -t VSMT
Traceback (most recent call last):
  File "/Users/noid/Library/Python/3.10/bin/vipaccess", line 5, in <module>
    from vipaccess.__main__ import main
  File "/Users/noid/Library/Python/3.10/lib/python/site-packages/vipaccess/__main__.py", line 10, in <module>
    from vipaccess import provision as vp
  File "/Users/noid/Library/Python/3.10/lib/python/site-packages/vipaccess/provision.py", line 34, in <module>
    from Crypto.Cipher import AES
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/__init__.py", line 27, in <module>
    from Crypto.Cipher._mode_ecb import _create_ecb_cipher
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_mode_ecb.py", line 35, in <module>
    raw_ecb_lib = load_pycryptodome_raw_lib("Crypto.Cipher._raw_ecb", """
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/_raw_api.py", line 309, in load_pycryptodome_raw_lib
    raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Not found '_raw_ecb.cpython-310-darwin.so', Cannot load '_raw_ecb.abi3.so': dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_raw_ecb.abi3.so, 0x0006): tried: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_raw_ecb.abi3.so' (no such file), '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_raw_ecb.abi3.so' (no such file), '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), Not found '_raw_ecb.so'

It appears python as shipped by Apple does not have the correct native code... I spent a lot of time researching issues with pycryptodome and M1 chips.. Finally installed pycryptodome with this command:

:/Users/noid
-> pip install pycryptodome --no-cache-dir --verbose --user  --force-reinstall
Using pip 22.3 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip (python 3.10)
Collecting pycryptodome
  Downloading pycryptodome-3.19.1-cp35-abi3-macosx_10_9_universal2.whl (2.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 6.9 MB/s eta 0:00:00
Installing collected packages: pycryptodome
Successfully installed pycryptodome-3.19.1

Since the install seemed to work, I then reinstalled python-vipaccess..

:/Users/noid
-> pip install python-vipaccess --no-cache-dir --verbose --user  --force-reinstall
Using pip 22.3 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip (python 3.10)
Collecting python-vipaccess
  Downloading python_vipaccess-0.14.1-py3-none-any.whl (18 kB)
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 1.1 MB/s eta 0:00:00
Collecting oath>=1.4.1
  Downloading oath-1.4.4-py3-none-any.whl (13 kB)
Collecting pycryptodome>=3.6.6
  Downloading pycryptodome-3.19.1-cp35-abi3-macosx_10_9_universal2.whl (2.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 9.4 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.3.2-cp310-cp310-macosx_11_0_arm64.whl (120 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.4/120.4 kB 63.2 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.1.0-py3-none-any.whl (104 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.6/104.6 kB 34.0 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 37.3 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.6-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 47.1 MB/s eta 0:00:00
Installing collected packages: oath, urllib3, pycryptodome, idna, charset-normalizer, certifi, requests, python-vipaccess
  Attempting uninstall: oath
    Found existing installation: oath 1.4.4
    Uninstalling oath-1.4.4:
      Removing file or directory /Users/noid/Library/Python/3.10/lib/python/site-packages/oath-1.4.4.dist-info/
      Removing file or directory /Users/noid/Library/Python/3.10/lib/python/site-packages/oath/
      Successfully uninstalled oath-1.4.4
  Attempting uninstall: pycryptodome
    Found existing installation: pycryptodome 3.19.1
    Uninstalling pycryptodome-3.19.1:
      Removing file or directory /Users/noid/Library/Python/3.10/lib/python/site-packages/Crypto/
      Removing file or directory /Users/noid/Library/Python/3.10/lib/python/site-packages/pycryptodome-3.19.1.dist-info/
      Successfully uninstalled pycryptodome-3.19.1
  changing mode of /Users/noid/Library/Python/3.10/bin/normalizer to 755
    Attempting uninstall: python-vipaccess
    Found existing installation: python-vipaccess 0.14.1
    Uninstalling python-vipaccess-0.14.1:
      Removing file or directory /Users/noid/Library/Python/3.10/bin/vipaccess
      Removing file or directory /Users/noid/Library/Python/3.10/lib/python/site-packages/python_vipaccess-0.14.1.dist-info/
      Removing file or directory /Users/noid/Library/Python/3.10/lib/python/site-packages/vipaccess/
      Successfully uninstalled python-vipaccess-0.14.1
  changing mode of /Users/noid/Library/Python/3.10/bin/vipaccess to 755
Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 oath-1.4.4 pycryptodome-3.19.1 python-vipaccess-0.14.1 requests-2.31.0 urllib3-2.1.0

Invoke vipaccess...

:/Users/noid
-> vipaccess provision -p -t VSMT
Generating request...
Fetching provisioning response from Symantec server...
Getting token from response...
Decrypting token...
Checking token against Symantec server...
Credential created successfully:
    otpauth://totp/VIP%20Access:VSMT12345678?secret=UFGHW3WPMQJ72HROV7YKMNGMYX5HWHCD&digits=6&algorithm=SHA1&image=https%3A%2F%2Fraw.githubusercontent.com%2Fdlenski%2Fpython-vipaccess%2Fmaster%2Fvipaccess.png&period=30
This credential expires on this date: 2027-01-07T18:52:20.371Z

You will need the ID to register this credential: VSMT12345678

Voila! back in the game. Maybe in the future Apple will update its installed version of python and this won't be necessary. Until then, there is a work around.

Lastly, it appears you no longer add the issuer= string to the credentials... why not? I added "&issuer=Charles%20Schwab" to the string to get a recognizable icon on the Authy app and it worked.

dlenski commented 10 months ago

Lastly, it appears you no longer add the issuer= string to the credentials... why not? I added "&issuer=Charles%20Schwab" to the string to get a recognizable icon on the Authy app and it worked.

See #53 and my attempt to fix it in https://github.com/dlenski/python-vipaccess/commit/acf264e20d90b73bd9b3bec330ccbbc958dc4420

If Authy now needs a different hand-holding treatment than it did 3 years ago, please continue the discussion there.

dlenski commented 10 months ago

python-vipaccess installed with --user option

:raised_eyebrow: Are you sure you installed it correctly the first time around?

I am extremely skeptical, because the rest of your error messages show that Python is not attempting to use any user-installed pycryptodome, but instead is attempting to use a systemwide installation of the Crypto module, built for the wrong processor architecture to boot. :raised_eyebrow: :raised_eyebrow: 

> OSError: Cannot load native module 'Crypto.Cipher._raw_ecb':
…
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto
…
(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), Not found '_raw_ecb.so'

Whereas when you show your second attempt to install python-vipaccess in more detail, it does correctly install pycryptodome as a dependency, just as we need it to.

dlenski commented 10 months ago

It appears that something went seriously awry in the initial installation attempt (python-vipaccess was attempting to import a Crypto module that was both installed system-wide and for the wrong CPU architecture :dizzy_face:).

A subsequent installation attempt shows everything going fine.

So I don't know what went wrong here, but it doesn't appear to have anything at all to do with python-vipaccess.

Feel free to reopen if you think I missed something, @beaufort2015.

beaufort2015 commented 10 months ago

Hello Dan,

Let's start over.

I just upgraded to Mac OS 14.2.1 Sonoma and I uninstalled all the python packages that were installed with --user option... basically I have a clean python3 installation as shipped by Apple. Below is the set of commands that I issued to illustrate my problems with getting vipaccess to run. My commends are augmented with <<< Comment >>>.

But basically the problem boils down to the fact the version of pycryptodome shipped by Apple does not have the correct native code for running on the Apple M1 chip...

So here we go...

<<< First list all the packages that have been installed with pip install --user option.... There are none. >>>

:/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> ll total 0 drwxr-xr-x 2 beaufort2015 staff 64 Jan 16 12:23 . drwxr-xr-x 4 beaufort2015 staff 128 Jan 15 17:13 ..

<<< next list all the packages install by Apple for Python 3.10, you can see many of the prerequisite packages installed >>>

:/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> pip list Package Version

astroid 2.12.12 autopep8 2.0.0 cachetools 5.2.0 certifi 2022.9.24 charset-normalizer 2.1.1 dill 0.3.6 distlib 0.3.6 filelock 3.8.0 google-api-core 2.10.2 google-api-python-client 2.65.0 google-auth 2.14.0 google-auth-httplib2 0.1.0 google-auth-oauthlib 0.7.0 googleapis-common-protos 1.56.4 httplib2 0.21.0 idna 3.4 isort 5.10.1 lazy-object-proxy 1.8.0 mccabe 0.7.0 numpy 1.23.4 oauthlib 3.2.2 pbr 5.10.0 pep8 1.7.1 pip 22.3 platformdirs 2.5.2 protobuf 4.21.9 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycodestyle 2.9.1 pycryptodome 3.15.0 pylint 2.15.5 pyparsing 3.0.9 requests 2.28.1 requests-oauthlib 1.3.1 rsa 4.9 setuptools 63.2.0 six 1.16.0 stevedore 4.0.0 tomli 2.0.1 tomlkit 0.11.6 uritemplate 4.1.1 urllib3 1.26.12 virtualenv 20.16.5 virtualenv-clone 0.5.7 virtualenvwrapper 4.8.4 wheel 0.37.1 wrapt 1.14.1 z3-solver 4.11.2.0 zxcvbn 4.4.28

[notice] A new release of pip available: 22.3 -> 23.3.2 [notice] To update, run: python3 -m pip install --upgrade pip

<<< Now install python-vipaccess with --user option :/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> pip install python-vipaccess --no-cache-dir --verbose --user Using pip 22.3 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip (python 3.10) Collecting python-vipaccess Downloading python_vipaccess-0.14.1-py3-none-any.whl (18 kB) Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from python-vipaccess) (2.28.1) Requirement already satisfied: pycryptodome>=3.6.6 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from python-vipaccess) (3.15.0) Collecting oath>=1.4.1 Downloading oath-1.4.4-py3-none-any.whl (13 kB) Requirement already satisfied: idna<4,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests->python-vipaccess) (3.4) Requirement already satisfied: charset-normalizer<3,>=2 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests->python-vipaccess) (2.1.1) Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests->python-vipaccess) (2022.9.24) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests->python-vipaccess) (1.26.12) Installing collected packages: oath, python-vipaccess Creating /Users/beaufort2015/Library/Python/3.10/bin changing mode of /Users/beaufort2015/Library/Python/3.10/bin/vipaccess to 755 Successfully installed oath-1.4.4 python-vipaccess-0.14.1

[notice] A new release of pip available: 22.3 -> 23.3.2 [notice] To update, run: python3 -m pip install --upgrade pip

<<< Now look and see what actually got installed >>>

:/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> ll total 0 drwxr-xr-x 6 beaufort2015 staff 192 Jan 16 12:26 . drwxr-xr-x 4 beaufort2015 staff 128 Jan 15 17:13 .. drwxr-xr-x 9 beaufort2015 staff 288 Jan 16 12:26 oath drwxr-xr-x 8 beaufort2015 staff 256 Jan 16 12:26 oath-1.4.4.dist-info drwxr-xr-x 10 beaufort2015 staff 320 Jan 16 12:26 python_vipaccess-0.14.1.dist-info drwxr-xr-x 8 beaufort2015 staff 256 Jan 16 12:26 vipaccess

<<< Now invoke vipaccess and it fails because of the missing native code

:/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> vipaccess provision -p -t VSMT Traceback (most recent call last): File "/Users/beaufort2015/Library/Python/3.10/bin/vipaccess", line 5, in

from vipaccess.__main__ import main File "/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages/vipaccess/__main__.py", line 10, in from vipaccess import provision as vp File "/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages/vipaccess/provision.py", line 34, in from Crypto.Cipher import AES File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/__init__.py", line 27, in from Crypto.Cipher._mode_ecb import _create_ecb_cipher File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_mode_ecb.py", line 35, in raw_ecb_lib = load_pycryptodome_raw_lib("Crypto.Cipher._raw_ecb", """ File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/_raw_api.py", line 309, in load_pycryptodome_raw_lib raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts))) OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Not found '_ raw_ecb.cpython-310-darwin.so', Cannot load '_raw_ecb.abi3.so': dlopen(/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_ raw_ecb.abi3.so, 0x0006): tried: '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_ raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_ raw_ecb.abi3.so' (no such file), '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Util/../Cipher/_ raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_ raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_ raw_ecb.abi3.so' (no such file), '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/Crypto/Cipher/_ raw_ecb.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), Not found '_raw_ecb.so' <<< Now lets simply force an install of pycryptodome to a more current level >>> :/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> pip install pycryptodome --no-cache-dir --verbose --user --force-reinstall Using pip 22.3 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip (python 3.10) Collecting pycryptodome Downloading pycryptodome-3.20.0-cp35-abi3-macosx_10_9_universal2.whl (2.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 8.9 MB/s eta 0:00:00 Installing collected packages: pycryptodome Successfully installed pycryptodome-3.20.0 [notice] A new release of pip available: 22.3 -> 23.3.2 [notice] To update, run: python3 -m pip install --upgrade pip <<< Now lets reinvoke vipaccess... and voila! it works. >>> :/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> vipaccess provision -p -t VSMT Generating request... Fetching provisioning response from Symantec server... Getting token from response... Decrypting token... Checking token against Symantec server... Credential created successfully: otpauth://totp/VIP%20Access:VSMT54723834?secret=FZNMQ6SWLFRBFJINWGXIHAN3RWYWPEDP&digits=6&algorithm=SHA1&image=https%3A%2F% 2Fraw.githubusercontent.com %2Fdlenski%2Fpython-vipaccess%2Fmaster%2Fvipaccess.png&period=30 This credential expires on this date: 2027-01-15T17:28:31.829Z You will need the ID to register this credential: VSMT54723834 You can use oathtool to generate the same OTP codes as would be produced by the official VIP Access apps: oathtool -b --totp FZNMQ6SWLFRBFJINWGXIHAN3RWYWPEDP # output one code oathtool -v -b --totp FZNMQ6SWLFRBFJINWGXIHAN3RWYWPEDP # ... with extra information <<< Lastly, let's look at what is installed in this user's python library >>> :/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> ll total 0 drwxr-xr-x 8 beaufort2015 staff 256 Jan 16 12:28 . drwxr-xr-x 4 beaufort2015 staff 128 Jan 15 17:13 .. drwxr-xr-x 16 beaufort2015 staff 512 Jan 16 12:28 Crypto drwxr-xr-x 9 beaufort2015 staff 288 Jan 16 12:26 oath drwxr-xr-x 8 beaufort2015 staff 256 Jan 16 12:26 oath-1.4.4.dist-info drwxr-xr-x 10 beaufort2015 staff 320 Jan 16 12:28 pycryptodome-3.20.0.dist-info drwxr-xr-x 10 beaufort2015 staff 320 Jan 16 12:26 python_vipaccess-0.14.1.dist-info drwxr-xr-x 8 beaufort2015 staff 256 Jan 16 12:26 vipaccess :/Users/beaufort2015/Library/Python/3.10/lib/python/site-packages -> At the end of the day, vipaccess works fine on a Mac running the Apple M1 chip instead of the Intel chip... but only if you upgrade the pycryptodome package. Level 3.20.0 works but level 3.15.0, which is shipped, doesn't. Finally, I was glad to see that Authy was able to load the Charles Schwab icon by adding "&issuer=Charles%20Schwab" to the otpauth string. Regards Mike On Tue, Jan 9, 2024 at 6:45 PM Dan Lenski ***@***.***> wrote: > It appears that something went seriously awry in the initial installation > attempt (python-vipaccess was attempting to import a Crypto module that > was both installed system-wide *and* for the wrong CPU architecture 😵). > > A subsequent installation attempt shows everything going fine. > > So I don't know what went wrong here, but it doesn't appear to have > anything at all to do with python-vipaccess. > > Feel free to reopen if you think I missed something, @beaufort2015 > . > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >