search

AntonKueltz / fastecdsa

Python library for fast elliptic curve crypto
https://pypi.python.org/pypi/fastecdsa
The Unlicense
264 stars 77 forks source link

Issues with getting fastecdsa working in Windows #11

Open ymgve opened 6 years ago

ymgve commented 6 years ago

There are some issues getting the module working in Windows - I found fixes, but they aren't very elegant:

AntonKueltz commented 6 years ago

Yes, I suppose it's not explicitly stated in the README (which it probably should be), but the package isn't written with Windows in mind as one of the target OSes. This is precisely because compiling the C extensions on Windows is a major pain, the main issues which you've touched on already (installing GMP on Windows is a nightmare and the C compiler for Python isn't anything to write home about either).

Does MPIR expose the same functions / interface that GMP does or did you have to rewrite all the GMP library calls to MPIR calls?

ymgve commented 6 years ago

MPIR was basically a drop-in replacement - just have to rename mpir.h to gmp.h and mpir.lib to gmp.lib in when you place them in your include/lib directories.

AntonKueltz commented 6 years ago

Understood. I can try to see if I can make the configuration / setup process play nice with Windows, but I can't promise any results. Ideally it would be nice to be able to just pip install fastecdsa on Windows without any further hassle but that seems a bit unrealistic given my experience writing packages for Windows. I'll keep the issue open until a satisfactory resolution is arrived at.

shikuk commented 6 years ago

I try this steps: 1) Download from http://www.mpir.org/downloads.html MPIR-2.6.0.source.bz2, unpack to MPIR-2.6.0 2) Download from http://yasm.tortall.net/Download.html Win32 .exe (for general use on 32-bit Windows) (Not shure it needed) rename it to yasm.exe and put to C:\Users\%username%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\bin 2) Run cmd.exe, cd to unpacked MPIR-2.6.0 dir 3) run "C:\Users\%username%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\vcvarsall.bat" so set Visual C++ for Python\9.0 compiler 4) cd to unpacked MPIR-2.6.0\win dir 5) edit configure.bat add 

set ABI=32
set VCTARGET=x86

between :exitlp
and     :: ARCH is native ABI
save and run configure.bat

6) run make.bat no errors/warnings 7) /optional/ run make.bat check -> OK (make tune and make speed fails) 8) run gen_mpir_h.bat 9) copy MPIR-2.6.0\gmp.h to C:\Users\%username%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include\gmp.h copy(whith rename) MPIR-2.6.0\win\mpir.lib C:\Users\%username%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\lib\gmp.lib

Well, first part done

as ymgve say,

I had to move all variable declarations to the start of functions

after this process python setup.py build -> builded OK, but have linker "warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library" python setup.py install check at my project. It works!

PaulGregor commented 5 years ago

Here is pre compiled versions for Windows https://github.com/ChillMagic/MPIR-Binary

AntonKueltz commented 5 years ago

Able to build and install but something is still going wrong -

$ python setup.py build
$ python setup.py install
$ python setup.py test
running test
Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\A\Desktop\fastecdsa\fastecdsa\test.py", line 13, in <module>
    from .ecdsa import sign, verify
  File "fastecdsa\ecdsa.py", line 4, in <module>
    from fastecdsa import _ecdsa
ImportError: DLL load failed: The specified module could not be found.

Continuing to investigate... (as you can probably tell I don't do much development on the Windows platform so if I'm missing something obvious please let me know)

shikuk commented 5 years ago

Probably, "setup.py install" doesn't pack _ecdsa.pyd to egg file or egg file not yet in Lib\site-packages\ I will try to reproduce steps at fresh windows install to refresh my memory

AntonKueltz commented 5 years ago

I'm getting the .pyd files in the install, and indeed ipython recognizes that e.g. curvemath is an importable module, but when I try to import it the DLL load fails. I presume this is because it correctly sees that there is no underlying DLL for curvemath or _ecdsa. I have .exp and .lib and .obj files but I seem to be missing the bin/ folder that would hold the DLLs.

shikuk commented 5 years ago

As I checkout my working machine, _ecdsa.dll not used at all. I have this file only at build dir and after removing this folder, scripts works as usual. If I remove _ecdsa.pyd from Lib\site-packages\fastecdsa-1.6.4-py2.7-win32.egg (7zip helps) scripts fails with File "d:\Python27\lib\site-packages\pkg_resources__init__.py", line 1883, in _extract_resource timestamp, size = self._get_date_and_size(self.zipinfo[zip_path]) KeyError: 'fastecdsa\_ecdsa.pyd' So, it looks like .dll have to be converted to .pyd and .pyd packed to egg

PaulGregor commented 5 years ago

I had the same error. To fix it, just put mpir.dll from MPIR-Binary-master\dll\x64\Release in to the directory of package, C:\Python37\Lib\site-packages\fastecdsa

shikuk commented 5 years ago

Here is somthing interesting - I have not directory of package, .\Lib\site-packages\fastecdsa just egg file in .\Lib\site-packages\ Different python build/install systems? How to check it? Also, no mpir.dll at my machine at all. Have mpir.lib at MPIR build dir copyed to VC\lib\gmp.lib Attached file is build log from scratch. But, from working machine, were exist working fastecdsa pack build.log

PaulGregor commented 5 years ago

Here is somthing interesting - I have not directory of package, .\Lib\site-packages\fastecdsa just egg file in .\Lib\site-packages Different python build/install systems? How to check it? Also, no mpir.dll at my machine at all. Have mpir.lib at MPIR build dir copyed to VC\lib\gmp.lib Attached file is build log from scratch. But, from working machine, were exist working fastecdsa pack build.log

You can just copy fastecdsa from egg directory to site-packages. Also I dont know about python 2.7. I've tested on 3+ only. My steps were: copy gmp.h to C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include copy mpir.lib to package directory, it could be C:\Python37\Lib\site-packages\fastecdsa or C:\Python37\Lib\site-packages\fastecdsa-1.6.5-py3.7-win-amd64.egg\fastecdsa

namuyan commented 5 years ago

Hello, I encountered same problem, but solved. I generate gmp.h, gmp.lib, gmp.def and gmp.dll from GMP. I check what's name of DLL lack by DependencyWalker. So I copy libgmp-10.dll from gmp.dll.

naphy0 commented 5 years ago

You need to have an microsoft visual 2019 and Microsoft c++ in visual studio to run the fastedcsa. You cannot install it without Microsoft c++. after that follow this instruction.

  1. Download https://github.com/ChillMagic/MPIR-Binary
  2. Download https://github.com/AntonKueltz/fastecdsa
  3. rename mpir.h to gmp.h from MPIR-Binary-master\include and rename mpir.lib to gmp.lib from MPIR-Binary-master\lib\MD\x64\Release
  4. Copy and paste gmp.lib C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.21.27702\lib\x64
  5. Copy and paste gmp.h C:\Phyton3\include
  6. I don't remember if I do this ( just put mpir.dll from MPIR-Binary-master\dll\x64\Release in to the directory of package, C:\Python37\Lib\site-packages\fastecdsa ) Do it last if you have error. if not working try to rename it again to gmp.dll But I guess it's not needed to install fastedcsa.
  7. Open CMD and cd to where you put your fastecdsa E:\Download\fastecdsa-master
  8. when your cmd is in the directory type: python setup.py build
  9. after installation type again: python setup.py install
  10. Good Luck treasure hunting.
ShadowJonathan commented 4 years ago

Per @naphy0's suggestion, i've (probably) gotten a build fully working by doing the following steps:

  1. Install Visual Studio 2019 buildtools, and adding the C++ suite from the main menu.
  2. Downloading MPIR-Library, and adding include/** to $pyinstall/include, and lib/MD/win32/Release to $pyinstall/libs
  3. Add extra_link_args=["/NODEFAULTLIB:MSVCRT"] to every extension object in setup.py (since the linker was complaining about it)
  4. Drink some coffee while running python setup.py build
  5. Watch intensely while running python setup.py test

And all of that resulted into a successful test.

Ran 32 tests in 9.355s

OK
ShadowJonathan commented 4 years ago

I pushed a commit to my fork that has the prerequisites all ready for a windows build, the only thing that needs to be installed is buildtools from here, under "Tools for Visual Studio 2019". (plus the C++ build tools)

https://github.com/ShadowJonathan/fastecdsa-any/commit/041d9f4aecc8461ef75df290b3289f15a2b39d41

If any contributor from here can please pull on a windows machine, build, and confirm the library works as intended, please let me know. This library is required in libp2p, and currently any windows build will fail because of a fastecdsa win32 wheel missing from the pypi repo (see https://github.com/libp2p/py-libp2p/issues/363).

ShadowJonathan commented 4 years ago

@AntonKueltz

AntonKueltz commented 4 years ago

Im currently on travel but will verify on my Windows VM as soon as I am able. Thanks for your work toward getting this package building on Windows!

ShadowJonathan commented 4 years ago

I've also been poking around looking at automated wheel builds for windows in general, this setup has worked on my normal desktop computer (due to some other options not wanting to work before), so I am not 100% sure either about this configuration working all the time, nor if the produced library is stable enough for use.

Thanks for your work toward getting this package building on Windows!

No problem!

ShadowJonathan commented 4 years ago

I've confirmed that building it like this only works with 32-bits python installs, anything 64-bit (amd64) fails while spitting this:

...
curveMath.c
src/curveMath.c(81): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
src/curveMath.c(115): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
src/curveMath.c(116): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
...
   Creating library build\temp.win-amd64-3.8\Release\src\curvemath.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\src\curvemath.cp38-win_amd64.exp
gmp.lib(mulmod_2expm1.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(mulmid.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(mulmid_n.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(toom42_mulmid.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(mulmod_2expp1.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(inv_divappr_q_n.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(dc_divappr_q.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(hgcd_appr.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(dc_bdiv_q.obj) : error LNK2001: unresolved external symbol __security_check_cookie
gmp.lib(hgcd_step.obj) : error LNK2001: unresolved external symbol __security_check_cookie
(etc.)

(this is after adding extra libraries under lib/MD/x64 and placing their folders first)

ShadowJonathan commented 4 years ago

I changed all libraries from MD to MT ones, and added a if-else chain that looks at distutils.utils.get_platform() to automatically add the right library dirs.

32-bit and 64-bit compile now (on two windows machines), the 64-bit one spits many warnings that're probably interesting to look at, but other than that, tests for both succeed.

(side-note, just changing the libraries key in the extension objects from gmp to mpir makes building and testing work, no change needed to the header files)

AntonKueltz commented 4 years ago

Was able to get your branch to build locally (only with python3, the compiler python2 wants is quite old). Since python2 is EOL in less than a month I will fold this into the 2.0 release that will only support python3 and make some backwards incompatible changes. Will also build and distribute wheels for ease. Will have to figure out how to get around bundling the GMP / MPIR code since having it in the repo will be incompatible with the public domain license.

Thanks for everyone's efforts towards getting this working on Windows, it took some time but we're almost there.

ShadowJonathan commented 4 years ago

All of my efforts are in https://github.com/ShadowJonathan/fastecdsa-any/tree/win32 and instructions on how to make it work are this and this comment (disregard the commit link in the last comment)

I've only bundled MPIR with that branch to quickly test it on multiple machines, the exact instructions on where which files come from are in my previous comments, the actual repository where the binary files are stored is https://github.com/ChillMagic/MPIR-Binary

I've managed to build this on 3.7-32-bit, 3.8-32-bit, and 3.8-64-bit, altough all builds worked and tested (successfully), the last build gave a few warnings.

I've not been able to discern any stability issues whatsoever, but that doesn't mean they cannot be there, I've never tested the builds beyond the default test suite.

ShadowJonathan commented 4 years ago

(oh whoops, did not read message in its entirety)

AntonKueltz commented 4 years ago

Yeah I think the way forward is to update the documentation to state which steps need to be taken if you wish to build from source on Windows rather than via a wheel. The test suite has pretty good coverage so I'm not too worried about stability issues (beyond the known issues in the tracker here) but I suppose as more people use it on Windows things will be reported if they're there.

I should also clarify that I can't speak to 32 bit machines as I only have a 64 bit machine available to test with currently.

p1r473 commented 3 years ago

You need to have an microsoft visual 2019 and Microsoft c++ in visual studio to run the fastedcsa. You cannot install it without Microsoft c++. after that follow this instruction.

  1. Download https://github.com/ChillMagic/MPIR-Binary
  2. Download https://github.com/AntonKueltz/fastecdsa
  3. rename mpir.h to gmp.h from MPIR-Binary-master\include and rename mpir.lib to gmp.lib from MPIR-Binary-master\lib\MD\x64\Release
  4. Copy and paste gmp.lib C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.21.27702\lib\x64
  5. Copy and paste gmp.h C:\Phyton3\include
  6. I don't remember if I do this ( just put mpir.dll from MPIR-Binary-master\dll\x64\Release in to the directory of package, C:\Python37\Lib\site-packages\fastecdsa ) Do it last if you have error. if not working try to rename it again to gmp.dll But I guess it's not needed to install fastedcsa.
  7. Open CMD and cd to where you put your fastecdsa E:\Download\fastecdsa-master
  8. when your cmd is in the directory type: python setup.py build
  9. after installation type again: python setup.py install
  10. Good Luck treasure hunting.

I had to do an additional step: Copy and paste gmp.lib C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64

It was looking for gmp.lib there, not in C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.21.27702\lib\x64

wakiyamap commented 2 years ago

I hope it will be helpful for those who build with github actions. https://github.com/eqlabs/pathfinder/pull/389/files

george012 commented 1 year ago

Compilation of [gmp] is now very easy, and [mingw64 + msys2 + vcpkg] can be mixed with future tool chains. But [fastecdsa] still can't compile, very distressed

ShadowJonathan commented 1 year ago

Tbh at this point in time I would just recommend getting it working in GHA with cibuildwheel and call it a day

AntonKueltz commented 1 year ago

Building a wheel vi cibuildwheel for windows is non trivial because of the third party native libs that need to be set up. We can't use e.g. brew or apt-get to install GMP. FWIW on the latest main branch there is an integration with GHA that builds wheels for linux and macos, still ironing out the kinks there.

I did manually build a wheel for tag v2.3.0 and CPython3.10 that can be downloaded here or via pip install fastecdsa==2.3.0 (note you have to be using a CPython3.10 interpreter / pip for the wheel to be used). Was able to install and run benchmark successfully on x64 Windows 10 locally. Is anyone able to verify that this wheel works for them as well?

Steps I used are what @naphy0 recommended above -

AntonKueltz commented 1 year ago

The trickiest part about building locally IMO is getting GMP into a directory that the C compiler / linker has on its path when it builds the project. I don't know enough about windows wheels to be able to say if wheels solve this problem by normalizing the path for the native third party libs. If someone who is not able to build the project from source could check if the wheel solves their issues that would be appreciated.

It seems like adding additional /LIBATH:<path>/gmp.lib arguments for the linker and -I<path>/gmp.h arguments for the compiler would also do the trick if we want those files in nonstandard locations, but again, Windows isn't my usual development OS so I'm not sure what the best way to get these passed to the build tools would be.

ShadowJonathan commented 1 year ago

We can't use e.g. brew or apt-get to install GMP.

...why is that a problem? You can use choco / winget if it's stored there, or manually download and run the installer from the command line (yes, it's possible if it's based off of installshield and the likes), or just hack around and store files in the correct locations.

It only needs to be reproducible once for it to be reproducible forever after, I highly recommend looking into it.

AntonKueltz commented 1 year ago

Are any of those tools available on the build hosts that cibuildwheel uses? Feel free to create a PR to update the existing build config to do what you recommended above. Again, Windows isn't my standard development OS, and while I do want to support Windows developers in this package I'm not the best person to implement the tooling.

ShadowJonathan commented 1 year ago

I might make a PR, I'll add it to my todo list

ShadowJonathan commented 1 year ago

Update: I've taken a look at the package manager options, and so far only vcpkg seems to support a simple install, according to this SO answer

george012 commented 1 year ago

更新：根据这个 SO answer，我查看了包管理器选项，到目前为止，似乎只有 vcpkg 支持简单安装 Just take a look, when vcpkg is successfully installed gmp, and see how to make fastecdsa support windows

ShadowJonathan commented 1 year ago

I've come basically as far as this CI run, with this workfile, but it gets stuck on some building error/problems wrt it not finding __imp____gmpz_clears and the likes.

if anyone can take a look that'd be amazing

AntonKueltz commented 1 year ago

Taking a look, it seems like it's able to find the GMP headers but not the library. The output is showing that the object files are being created, but it appears linking is failing when it tries to resolve the GMP symbols.