crypto: speed up secp256k1/field.py

[teknico]

This speeds up most of crypto/secp256k1/field.py by ~100x (!) using Cython.

The file crypto/secp256k1/field.pxd contains type definitions that are used by Cython to generate field.c, a 23K lines C extension that in turn gets compiled to a dynamic library in a platform-dependent format (.so on Unix-like, .dll on Windows).

The dynamic library is placed in the same directory and is loaded automatically by the Python interpreter at execution time.

The command to generate the C extension and build the dynamic library in development is:

cythonize -X language_level=3 -a -i ./decred/crypto/secp256k1/field.py

Poetry has undocumented support for building C extensions, so that the poetry build command builds the library and includes it into the created, platform-dependent, installable wheel archive.

Cython has been added to the development dependencies of the decred library.

Only tested on Linux, let me know if it works correctly on Windows and macOs.

Based on #158.

[teknico]

This is an amazing improvement.

Thank you.

1. Should we be checking in our .c files?

I don't think so. From the docs:

It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.

So the user is supposed to have a C compiler but somehow not Cython, that is listed in the dev dependencies? It doesn't seem likely.

It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he/she probably doesn’t want to use it just to install your module.

Huh? Why should they not want to use it?

Also, the installed version may not be the same one you used, and may not compile your sources correctly.

We have dependency check and resolution now, thanks to Poetry. This line of reasoning is outdated.

Then they go on detailing lots of unneeded complexity. No thanks.

2. There are some official recommendations for distributing though PyPI. Does Poetry take care of all of this?

Yes, it does. Just run poetry build and you have your platform-dependent wheel archive, ready to be uploaded by poetry publish. That is all.

[buck54321]

I tried to poetry install on a clean install of Windows 10, and I got

[EnvCommandError]
Command ['C:\\Users\\buck5\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\decred-81_y65CL-py3.8\\Scripts\\pip.exe', 'install', '-e', 'C:\\Users\\buck5\\programs\\tinydecred-secp256k1-field-speedup\\tinydecred-secp256k1-field-speedup\\decred'] errored with the following return code 1, and output:
Obtaining file:///C:/Users/buck5/programs/tinydecred-secp256k1-field-speedup/tinydecred-secp256k1-field-speedup/decred
Requirement already satisfied: appdirs<2.0.0,>=1.4.3 in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from decred==0.0.1) (1.4.3)
Requirement already satisfied: base58<3.0.0,>=2.0.0 in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from decred==0.0.1) (2.0.0)
Requirement already satisfied: blake256<0.2.0,>=0.1.1 in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from decred==0.0.1) (0.1.1)
Requirement already satisfied: pynacl<2.0.0,>=1.3.0 in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from decred==0.0.1) (1.3.0)
Requirement already satisfied: websocket_client<0.58.0,>=0.57.0 in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from decred==0.0.1) (0.57.0)
Requirement already satisfied: six in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from pynacl<2.0.0,>=1.3.0->decred==0.0.1) (1.14.0)
Requirement already satisfied: cffi>=1.4.1 in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from pynacl<2.0.0,>=1.3.0->decred==0.0.1) (1.14.0)
Requirement already satisfied: pycparser in c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\lib\site-packages (from cffi>=1.4.1->pynacl<2.0.0,>=1.3.0->decred==0.0.1) (2.20)
Installing collected packages: decred
  Running setup.py develop for decred
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\buck5\\programs\\tinydecred-secp256k1-field-speedup\\tinydecred-secp256k1-field-speedup\\decred\\setup.py'"'"'; __file__='"'"'C:\\Users\\buck5\\programs\\tinydecred-secp256k1-field-speedup\\tinydecred-secp256k1-field-speedup\\decred\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: C:\Users\buck5\programs\tinydecred-secp256k1-field-speedup\tinydecred-secp256k1-field-speedup\decred\
    Complete output (11 lines):
    running develop
    running egg_info
    writing decred.egg-info\PKG-INFO
    writing dependency_links to decred.egg-info\dependency_links.txt
    writing requirements to decred.egg-info\requires.txt
    writing top-level names to decred.egg-info\top_level.txt
    reading manifest file 'decred.egg-info\SOURCES.txt'
    writing manifest file 'decred.egg-info\SOURCES.txt'
    running build_ext
    building 'decred.crypto.secp256k1.field' extension
    error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\buck5\appdata\local\pypoetry\cache\virtualenvs\decred-81_y65cl-py3.8\scripts\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\buck5\\programs\\tinydecred-secp256k1-field-speedup\\tinydecred-secp256k1-field-speedup\\decred\\setup.py'"'"'; __file__='"'"'C:\\Users\\buck5\\programs\\tinydecred-secp256k1-field-speedup\\tinydecred-secp256k1-field-speedup\\decred\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

If there is no compiler, it must fail gracefully and fall back to pure-Python.

[teknico]

If there is no compiler, it must fail gracefully and fall back to pure-Python.

Agreed. This needs Poetry support though, so I created an issue in their project.

[buck54321]

Here's a build.py that works.

https://github.com/buck54321/tinydecred/blob/teknico/decred/build.py

[teknico]

Here's a build.py that works.

That's swell! Thanks, putting it in.