search

AntonKueltz / fastecdsa

Python library for fast elliptic curve crypto
https://pypi.python.org/pypi/fastecdsa
The Unlicense
263 stars 76 forks source link

Cannot install in macOS BigSur (M1 chip) #74

Open martriay opened 2 years ago

martriay commented 2 years ago

I'm trying to install fastecdsa on macOS BigSur (M1 chip) by running

(venv) $ pip3 install fastecdsa

and even though I previously installed gmp:

$ brew install gmp

it cant find the lib, no matter what I do

    src/curve.h:4:10: fatal error: 'gmp.h' file not found
    #include "gmp.h"

Although the error changed when I created a symlink

ln -s /opt/homebrew/include/gmp.h /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include

and now I get:

    ld: library not found for -lgmp
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/bin/gcc' failed with exit code 1

I also tried:

I can't put my finger on it :(

milancermak commented 2 years ago

This worked for me:

CFLAGS=-I/opt/homebrew/opt/gmp/include LDFLAGS=-L/opt/homebrew/opt/gmp/lib pip install fastecdsa

AntonKueltz commented 2 years ago

@martriay did the above work for you? If you have installed gmp via brew you should be able to point the compiler to the correct headers and libs via CFLAGS and LDFLAGs. You can use brew --prefix to determine the root path of your brew installed packages if they are different from the above comment.

martriay commented 2 years ago

@AntonKueltz no, but I compiled a list of solutions to my specific problem: https://github.com/OpenZeppelin/nile/issues/22#issuecomment-945039043

justinba1010 commented 2 years ago

Hi @martriay, I think this should work. I just tested it on my M1 mac. python3 -m pip install --no-binary :all: --no-use-pep517 fastecdsa I've been using this as a pip alias for awhile now. Give this a shot and let me know.

martriay commented 2 years ago

Hi @martriay, I think this should work. I just tested it on my M1 mac. python3 -m pip install --no-binary :all: --no-use-pep517 fastecdsa I've been using this as a pip alias for awhile now. Give this a shot and let me know.

Even with gmp installed:

(cairo_venv) ➜  ~ brew reinstall gmp
==> Downloading https://ghcr.io/v2/homebrew/core/gmp/manifests/6.2.1_1
Already downloaded: /Users/martriay/Library/Caches/Homebrew/downloads/a1986a80dafb4e42b8372268a00d8d14aa8294154c876ba82e7ce48db06885cb--gmp-6.2.1_1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/gmp/blobs/sha256:491220f1ff2c662b96295d931a80702523eeaee681d7305fb02b561e527dcbb8
Already downloaded: /Users/martriay/Library/Caches/Homebrew/downloads/3e74dcc39d8faf31f1e226060beea4bda63ec6888f7fe71d3513fd2b977df55e--gmp--6.2.1_1.arm64_big_sur.bottle.tar.gz
==> Reinstalling gmp
==> Pouring gmp--6.2.1_1.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/gmp/6.2.1_1: 21 files, 3.3MB
gcc -bundle -undefined dynamic_lookup -arch x86_64 -g build/temp.macosx-10.9-x86_64-3.9/src/curve.o build/temp.macosx-10.9-x86_64-3.9/src/curveMath.o build/temp.macosx-10.9-x86_64-3.9/src/point.o -lgmp -o build/lib.macosx-10.9-x86_64-3.9/fastecdsa/curvemath.cpython-39-darwin.so
    ld: library not found for -lgmp
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/martriay/cairo_venv/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/m3/c_rs0zls4bg6k2d328r293kr0000gn/T/pip-install-5v_4xzlj/fastecdsa_c2411a6e2b5c4ee6af8fde06bf939490/setup.py'"'"'; __file__='"'"'/private/var/folders/m3/c_rs0zls4bg6k2d328r293kr0000gn/T/pip-install-5v_4xzlj/fastecdsa_c2411a6e2b5c4ee6af8fde06bf939490/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/m3/c_rs0zls4bg6k2d328r293kr0000gn/T/pip-record-flm9jcii/install-record.txt --single-version-externally-managed --compile --install-headers /Users/martriay/cairo_venv/include/site/python3.9/fastecdsa Check the logs for full command output.

I'm honestly a bit tired of trying. I have it running on a dockerized ubuntu, it's not perfect but at least I'm not wasting more time with this.

AntonKueltz commented 2 years ago

Sorry to hear that this is still broken for you on the M1 chips. I have no way to repro so I'm afraid I can't help much here. The only thing I can think of is that GMP appears to be installing for the ARM architecture (Pouring gmp--6.2.1_1.arm64_big_sur.bottle.tar.gz), whereas GCC is compiling against the x86 architecture (-arch x86_64). This seems bound to cause issues.

martriay commented 2 years ago

@AntonKueltz thanks for trying! I have a good-enough solution at the moment (spinning up a docker), so I'm good. Let's hope for this to work in the future.

justinba1010 commented 2 years ago

Hi @martriay,

Ahh, I think I fixed that for another library. So the fix for that gmp issue is,

arch -arm64 brew install gmp

Hope that helps!

justinba1010 commented 2 years ago

I would go ahead and alias brew install to arch -arm64 brew install, I've made the mistake of forgetting too many times to count.

Sorry for the delay as well, I don't really use github too much as I no longer use it for work.

martriay commented 2 years ago

Thanks. I did a bit of everything proposed here by using arm64, compiler flags, etc:

arch -arm64 brew install gmp

CFLAGS=-I/opt/homebrew/opt/gmp/include LDFLAGS=-L/opt/homebrew/opt/gmp/lib python3 -m pip install --no-binary :all: --no-use-pep517 fastecdsa

And although installation succeeded, execution fails:

  File "/Users/martriay/dev/openzeppelin/cairo-contracts/env/lib/python3.9/site-packages/fastecdsa/point.py", line 1, in <module>
    from fastecdsa import curvemath
ImportError: dlopen(/Users/martriay/dev/openzeppelin/cairo-contracts/env/lib/python3.9/site-packages/fastecdsa/curvemath.cpython-39-darwin.so, 2): Symbol not found: ___gmpz_add
  Referenced from: /Users/martriay/dev/openzeppelin/cairo-contracts/env/lib/python3.9/site-packages/fastecdsa/curvemath.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/martriay/dev/openzeppelin/cairo-contracts/env/lib/python3.9/site-packages/fastecdsa/curvemath.cpython-39-darwin.so
justinba1010 commented 2 years ago

Can you try without the CFLAGS and LDFLAGS? That and if that fails try reinstalling gmp and it's dependents. And trying without the flags. I think pip is smart enough to know how to compile these things if you install gmp with arm64 arch, and the flags may be overriding(also it looks like those flags are not necessarily where the gmp lib goes as for me it is:

/opt/homebrew/include/gmp.h
/opt/homebrew/Cellar/gmp/6.2.1_1/include/gmp.h

) So I would just get rid of those flags and try again.

martriay commented 2 years ago

Yes, it fails without them. Tried uninstalling gmp and creating a whole new venv before too.

    /usr/local/include/stdlib.h:359:13: note: insert '_Nonnull' if the pointer should never be null
    extern char *suboptarg;         /* getsubopt(3) external variable */
                ^
                  _Nonnull
    180 warnings generated.
    gcc -bundle -undefined dynamic_lookup -arch x86_64 -g build/temp.macosx-10.9-x86_64-3.9/src/curve.o build/temp.macosx-10.9-x86_64-3.9/src/curveMath.o build/temp.macosx-10.9-x86_64-3.9/src/point.o -lgmp -o build/lib.macosx-10.9-x86_64-3.9/fastecdsa/curvemath.cpython-39-darwin.so
    ld: library not found for -lgmp
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/martriay/dev/openzeppelin/cairo-contracts/env/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/m3/c_rs0zls4bg6k2d328r293kr0000gn/T/pip-install-212875o5/fastecdsa_e6c1c9f4740848e79325926bc03d88c2/setup.py'"'"'; __file__='"'"'/private/var/folders/m3/c_rs0zls4bg6k2d328r293kr0000gn/T/pip-install-212875o5/fastecdsa_e6c1c9f4740848e79325926bc03d88c2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/m3/c_rs0zls4bg6k2d328r293kr0000gn/T/pip-record-ov2rq6lk/install-record.txt --single-version-externally-managed --compile --install-headers /Users/martriay/dev/openzeppelin/cairo-contracts/env/include/site/python3.9/fastecdsa Check the logs for full command output.

I'm impressed with your perseverance.

jw122 commented 2 years ago

This worked for me:

CFLAGS=-I/opt/homebrew/opt/gmp/include LDFLAGS=-L/opt/homebrew/opt/gmp/lib pip install fastecdsa

This also worked for me, thank you @milancermak! I'm on macOS big sur 11.2.3

Mrtrl commented 2 years ago

This worked for me:

CFLAGS=-I/opt/homebrew/opt/gmp/include LDFLAGS=-L/opt/homebrew/opt/gmp/lib pip install fastecdsa

This also worked for me!! Thanks

Rohith-sreedharan commented 2 years ago

This worked Dude, thanks for that

justinba1010 commented 2 years ago

Can you add more details @Rohith-sreedharan? Might be helpful for those stumbling into this.

neilmi commented 1 year ago

hey guys, i had this exact same problem on mac monterey m1 the way i solved it:

  1. uninstalled python completely
  2. installed python3.x exactly like as this article says: https://diewland.medium.com/how-to-install-python-3-7-on-macbook-m1-87c5b0fcb3b5
  3. ibrew install gmp
  4. then it should work :)

@martriay

dhvanipa commented 1 year ago

This worked for me:

CFLAGS=-I/opt/homebrew/opt/gmp/include LDFLAGS=-L/opt/homebrew/opt/gmp/lib pip install fastecdsa

This didn't work for me on an Intel chip. Instead I had to change the path of gmp to where it actually is (ie where homebrew installs it)

CFLAGS=-I/usr/local/Cellar/gmp/6.2.1_1/include LDFLAGS=-L/usr/local/Cellar/gmp/6.2.1_1/lib pip install fastecdsa

AntonKueltz commented 1 year ago

Correct, the thread was specific to M1 architecture Macs, which installs to a different path. I recommend using brew --prefix to abstract away the install directory. For example -

$ echo "$(brew --prefix)/lib"
/opt/homebrew/lib
dolores-park commented 1 year ago

was able to make this work on M1, by

  1. ibrew install gmp as suggested by @neilmi
  2. python3 -m pip install --no-binary :all: fastecdsa as suggested by @justinba1010
b1aafulei commented 5 months ago

In case anyone gets a similar error on an Intel mac

      In file included from src/curve.c:1:
      src/curve.h:4:10: fatal error: 'gmp.h' file not found
      #include "gmp.h"
               ^~~~~~~
      1 error generated.
      error: command '/usr/local/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for fastecdsa

Here is how I solved it -

brew install gmp
# just to make sure gmp is linked
brew unlink gmp && brew link gmp
CFLAGS=-I/usr/local/Cellar/gmp/6.3.0/include LDFLAGS=-L/usr/local/Cellar/gmp/6.3.0/lib pip install fastecdsa
MarcoDotIO commented 2 months ago

For context, this was apart of my issue with installing bitcoinlib. I made sure the CFlag and LDFlag variables were set when install fastecdsa. However, when attempting to use fastecdsa, I get this error:

Traceback (most recent call last):
  File "/Volumes/OpenDive/mastering-bitcoin/HelloWorld.py", line 1, in <module>
    from bitcoinlib.wallets import Wallet
  File "/Users/marcodotio/Library/Python/3.12/lib/python/site-packages/bitcoinlib/__init__.py", line 22, in <module>
    import bitcoinlib.keys
  File "/Users/marcodotio/Library/Python/3.12/lib/python/site-packages/bitcoinlib/keys.py", line 33, in <module>
    from fastecdsa import _ecdsa
ImportError: dlopen(/Users/marcodotio/Library/Python/3.12/lib/python/site-packages/fastecdsa/_ecdsa.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '___gmpz_add'
AntonKueltz commented 1 month ago

Do you have a minimal example you can share, e.g. HelloWorld.py of how you're using bicoinlib (and by extension this library)? In general the _ecdsa package should not be used directly by external packages, the usage by the importing library may be causing issues given the way this library is currently set up.