search

cossacklabs / themis

Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.
https://www.cossacklabs.com/themis
Apache License 2.0
1.88k stars 143 forks source link

installation issue: can't find pythemis on macOS M1 with non-brew way of installing python #913

Closed AikakiAdmin closed 2 years ago

AikakiAdmin commented 2 years ago

Describe the bug I am trying to use Themis using python on MacOS I installed brew first and followed the instructions. but I am not able to run the code and following error is occurring everytime, what could be be cause of this behaviour.?

Error File "securecell.py", line 4, in master_key = GenerateSymmetricKey() File "/Users/aikaki/Documents/Python Programs/themis/venv/lib/python3.8/site-packages/pythemis/skeygen.py", line 71, in GenerateSymmetricKey res = themis.themis_gen_sym_key(None, byref(key_length)) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/init.py", line 378, in getattr func = self.getitem(name) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/init.py", line 383, in getitem func = self._FuncPtr((name_or_ordinal, self)) AttributeError: dlsym(RTLD_DEFAULT, themis_gen_sym_key): symbol not found

Expected behavior I was expecting the code to run smoothly

Environment (please complete the following information):

vixentael commented 2 years ago

thank you @AikakiAdmin, we will take a look on M1 installation.

maybe openssl installation path? 🤔 cc @ilammy

vixentael commented 2 years ago

@AikakiAdmin could you please provide step by step commands and their outputs that you are running?

We have tested the following on M1 with macOS 12.3.1 (21E258) according to the installation instructions.

Installing themis

brew tap cossacklabs/tap
brew install libthemis

==> Downloading https://github.com/cossacklabs/themis/archive/0.14.0.tar.gz
==> Downloading from https://codeload.github.com/cossacklabs/themis/tar.gz/refs/tags/0.14.0
                                                      #   -=#=- #   #         
==> Reinstalling cossacklabs/tap/libthemis 
==> make install
🍺  /opt/homebrew/Cellar/libthemis/0.14.0: 34 files, 671.8KB, built in 8 seconds
==> Running `brew cleanup libthemis`...
pip install pythemis

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting pythemis
  Downloading pythemis-0.14.0.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six in /opt/homebrew/lib/python3.9/site-packages (from pythemis) (1.16.0)
Building wheels for collected packages: pythemis
  Building wheel for pythemis (setup.py) ... done
  Created wheel for pythemis: filename=pythemis-0.14.0-py2.py3-none-any.whl size=15946 sha256=d7f3186ea4cc994246e5fb19b6cd163544d8a45ea4d88be10900252fcee4efd9
  Stored in directory: /Users/arturhil/Library/Caches/pip/wheels/54/14/6a/618dc12818f899bec8a0d7f1494f279a53329feae52da7ada2
Successfully built pythemis
Installing collected packages: pythemis
  DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed pythemis-0.14.0

Generating keys

python3

from pythemis.skeygen import GenerateSymmetricKey
master_key = GenerateSymmetricKey()
print(master_key)

b'\xcb\x1e\x1d\xca\x9dU\xca\x1b\xac\x94\xd7\x18\x81\xd6Fd\xc7\x84#R\x7f\xe0\xe2o\xf2\xbd\xca["\xd4v\x0e'

that's working on our side. Let's try to find the difference.

AikakiAdmin commented 2 years ago

aikaki@Vijays-MacBook-Pro documents % brew tap cossacklabs/tap ==> Auto-updated Homebrew! Updated 1 tap (homebrew/core). ==> Updated Formulae Updated 1 formula.

==> Tapping cossacklabs/tap Cloning into '/opt/homebrew/Library/Taps/cossacklabs/homebrew-tap'... remote: Enumerating objects: 89, done. remote: Counting objects: 100% (2/2), done. remote: Total 89 (delta 1), reused 1 (delta 1), pack-reused 87 Receiving objects: 100% (89/89), 20.62 KiB | 879.00 KiB/s, done. Resolving deltas: 100% (32/32), done. Tapped 1 formula (16 files, 49.9KB). aikaki@Vijays-MacBook-Pro documents % brew install libthemis ==> Downloading https://github.com/cossacklabs/themis/archive/0.14.0.tar.gz Already downloaded: /Users/aikaki/Library/Caches/Homebrew/downloads/35b0334bda822db50514b9a9aad69cc14d96a708105b0c31edefb8a9a6375e0b--themis-0.14.0.tar.gz ==> Installing libthemis from cossacklabs/tap ==> make install 🍺 /opt/homebrew/Cellar/libthemis/0.14.0: 34 files, 671.8KB, built in 3 seconds ==> Running brew cleanup libthemis... Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP. Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).aikaki@Vijays-MacBook-Pro documents % pip3 install pythemis
Defaulting to user installation because normal site-packages is not writeable Collecting pythemis Using cached pythemis-0.14.0.tar.gz (12 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: six in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages (from pythemis) (1.15.0) Building wheels for collected packages: pythemis Building wheel for pythemis (setup.py) ... done Created wheel for pythemis: filename=pythemis-0.14.0-py2.py3-none-any.whl size=15928 sha256=d0332faea884cf12778e0c62f2c531879b33147ec821baf3e57a152bd25f73f9 Stored in directory: /Users/aikaki/Library/Caches/pip/wheels/21/b0/0d/767b69bf9f45e0606ba0b04f8cea0e0bfef38f47a5b47c312b Successfully built pythemis Installing collected packages: pythemis

AikakiAdmin commented 2 years ago

It seems you installed later version of the python using brew. My python site package in not writeable

ilammy commented 2 years ago

Okay... so the library seems to be at least loaded, right?

Could you please do

import ctypes.util

print(ctypes.util.find_library('libthemis'))

which is how PyThemis locats the native library. This should print out the following path:

/opt/homebrew/lib/libthemis.dylib

(which is normally a symlink into homebrew's cellar)

If it's something else, this means Python is probably loading not the library it should load.

However, if it is that path, let's check the library symbols:

nm -gU /opt/homebrew/lib/libthemis.dylib

if there is no _themis_gen_sym_key there, then indeed it's not there.

At the moment I can't find any known issues in Python library loading & name resolution in ctypes, but this could very well be something linked to that.

AikakiAdmin commented 2 years ago

code is returning None, which mean it is not getting the library. so a path problem.

ikaki@Vijays-MacBook-Pro themis % nm -gU /opt/homebrew/lib/libthemis.dylib 0000000000001f74 T _secure_comparator_append_secret 0000000000001f90 T _secure_comparator_begin_compare 0000000000001eec T _secure_comparator_create 0000000000001f40 T _secure_comparator_destroy 0000000000002380 T _secure_comparator_get_result 00000000000020c4 T _secure_comparator_proceed_compare 0000000000004b78 T _secure_session_cleanup 0000000000005738 T _secure_session_connect 0000000000005088 T _secure_session_create 0000000000004bc8 T _secure_session_destroy 0000000000005108 T _secure_session_generate_connect_request 0000000000005b40 T _secure_session_get_remote_id 0000000000004bfc T _secure_session_init 0000000000005970 T _secure_session_is_established 0000000000006528 T _secure_session_load 0000000000005980 T _secure_session_receive 000000000000645c T _secure_session_save 0000000000005844 T _secure_session_send 00000000000060ec T _secure_session_unwrap 0000000000005fd8 T _secure_session_wrap 00000000000034bc T _themis_gen_ec_key_pair 00000000000033f8 T _themis_gen_rsa_key_pair 0000000000003688 T _themis_gen_sym_key 00000000000034d4 T _themis_get_asym_key_kind 0000000000003590 T _themis_is_valid_asym_key 0000000000001860 T _themis_secure_cell_decrypt_context_imprint 0000000000001594 T _themis_secure_cell_decrypt_seal 0000000000001740 T _themis_secure_cell_decrypt_seal_with_passphrase 0000000000001820 T _themis_secure_cell_decrypt_token_protect 0000000000001844 T _themis_secure_cell_encrypt_context_imprint 00000000000014ac T _themis_secure_cell_encrypt_seal 0000000000001644 T _themis_secure_cell_encrypt_seal_with_passphrase 00000000000017fc T _themis_secure_cell_encrypt_token_protect 0000000000003890 T _themis_secure_message_decrypt 00000000000036b8 T _themis_secure_message_encrypt 000000000000397c T _themis_secure_message_sign 0000000000003b9c T _themis_secure_message_unwrap 0000000000003a10 T _themis_secure_message_verify 0000000000003abc T _themis_secure_message_wrap aikaki@Vijays-MacBook-Pro themis %

AikakiAdmin commented 2 years ago

library is present

aikaki@Vijays-MacBook-Pro documents % cd /opt/homebrew/lib/ aikaki@Vijays-MacBook-Pro lib % ls libsoter.0.dylib libspatialindex.6.1.1.dylib libspatialindex_c.6.1.1.dylib libthemis.0.dylib pkgconfig libsoter.a libspatialindex.6.dylib libspatialindex_c.6.dylib libthemis.a libsoter.dylib libspatialindex.dylib libspatialindex_c.dylib libthemis.dylib aikaki@Vijays-MacBook-Pro lib %

AikakiAdmin commented 2 years ago

added path also but same output "None" for the code you suggested. Any idea how libraries are searched in Mac?

aikaki@Vijays-MacBook-Pro ~ % echo $PATH /opt/homebrew/bin:/opt/homebrew/sbin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/lib/

AikakiAdmin commented 2 years ago

This problem is resolved when installed python using brew. My confusion is if I need to package Themis in docker, what step I need in Dockerfile

aikaki@Vijays-MacBook-Pro themis % python3 securecell.py
/opt/homebrew/lib/libthemis.dylib

ilammy commented 2 years ago

Uh-huh... Thanks, noted your observation. I believe we should add some check for None later, so that this situation produces a more helpful error message. Or just hardcode some extra paths to search in, so that it "just works".

I guess, homebrew's Python has homebrew paths baked into it, while with non-brew Python you wouldn't get /opt/homebrew/lib in your search paths. On x86, I think, it kinda works out because /usr/local/lib is in standard search paths or something. So even if you install Python from a different source, it still works. But not on M1 where /opt in not in the defaults.

Any idea how libraries are searched in Mac?

There is DYLD_LIBRARY_PATH environment variable with extra search paths, Python should respect it. (Or LD_LIBRARY_PATH on Linux.)

You can also tweak the native library search paths via sys.path variable as well, IIRC.

ilammy commented 2 years ago

if I need to package Themis in docker, what step I need in Dockerfile

Like, with macOS in Docker?.. 🤔

If you use a Linux-based image for Docker, there are Themis packages for some distros (Fedora, CentOS, Ubuntu, Debian – not for Alpine though, that one you'd likely need to build from source).

If you use Python from the distro's package repositories, Themis packages should work with it fine: yum/apt/rpm them after adding Cossack Labs' repositories.

AikakiAdmin commented 2 years ago

I already built a docker image which uses fast api frame work and I want add some encryption as part of APIs. I am in love with Themis for its simplicity of use and light weight. Let me see how we can go to next step. For now issue is closed.

On Sat, 16 Apr 2022 at 7:21 PM, Oleksii Lozovskyi @.***> wrote:

if I need to package Themis in docker, what step I need in Dockerfile

Like, with macOS in Docker?.. 🤔

If you use a Linux-based image for Docker, there are Themis packages for some distros https://docs.cossacklabs.com/themis/installation/installation-from-packages/ (Fedora, CentOS, Ubuntu, Debian – not for Alpine though, that one you'd likely need to build from source).

If you use Python from the distro's package repositories, Themis packages should work with it fine: yum/apt/rpm them after adding Cossack Labs' repositories.

— Reply to this email directly, view it on GitHub https://github.com/cossacklabs/themis/issues/913#issuecomment-1100669088, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN67P53HABTJW3PN3MJL2TDVFLAXBANCNFSM5TQAMW2Q . You are receiving this because you were mentioned.Message ID: @.***>