search

pkel / cpr

consensus protocol research
8 stars 2 forks source link

Package Gym as Python wheel #31

Closed pkel closed 1 year ago

pkel commented 1 year ago

This PR adds infrastructure for building Python wheels, eventually enabling pushing the OpenAI Gym to PyPI.

  1. Build OCaml DLL from setup.py
  2. Build platform specific wheels which are reusable across python versions
  3. Automate with cibuildwheel
  4. Backup mechanism in case OCaml toolchain is not available
  5. Publish as Github release (mark as prerelease if version contains '+')
pkel commented 1 year ago

This seems to be doing what we want.

I've tested on an old MacBook Air and a couple of Linux machines. Ben tested on OSX too. Seems solid.

We lost the ability to update the engine with python -m cpr_gym --update. But that might be a good thing. Simple install strategy will be do install wheel from pypi. Advances users can either install opam and build locally or extract a prebuilt engine from the appropriate wheel.

Before publishing to pypi I should definitely clean up README and DEVELOPMENT. Maybe merge them.

pkel commented 1 year ago

Just tested this on pypy. Install works, but loading the DLL fails.

  File "/home/patrik/tmp/venv/lib/pypy3.8/site-packages/cpr_gym/__init__.py", line 5, in <module>
    from ctypes import PyDLL, RTLD_GLOBAL, c_char_p
ImportError: cannot import name 'PyDLL' from 'ctypes' (/nix/store/36r439kdxz0hbd260lxi92w704ld3sxa-pypy3-7.3.7/pypy3-c/lib-python/3/ctypes/__init__.py)
pkel commented 1 year ago

Pypy does not support PyDLL.

Pythonlib examples show different methods of compiling and loading the shared library. Some compile bytecode shared objects the other native shared objects. Some use PyDLL, the others import. Easiest solutions seems to be bytecode + import. I'm confused. My combination of native + PyDLL is not shown there. Maybe I got it from somewhere else. Makes me wonder why I chose native mode. Maybe it is faster? How much?

Edit: I start to understand. Compiling .bc.so did not work and does not work as expected. Dune complains about missing libraries during linking. Same issue is described here . I got my build mode from the last comment, probably.

pkel commented 1 year ago

Max is sceptical about my OCaml extensions being compatible with any Python version. I'm paraphrasing.

CPython ABI compatibility is hard. Only recently ABI3 was introduced to achieve forward compatibility. Getting ABI3 right seems to be not easy. And that's only CPython. The OCamlers are missing something. Have a look at abi3audit.

https://blog.trailofbits.com/2022/11/15/python-wheels-abi-abi3audit/

Just tried to run abi3audit against my wheels. Does not work because I have not tagged my wheels as ABI3. Building proper ABI3 wheels requires some effort. Also, ABI3 is CPython only. If OCaml extensions are indeed Python-independent, then ABI3 wheel would defeat the purpose.

So, I think I should do some testing first. Load my DLL from python2 and pre-ABI3 cpython3 versions. Maybe also try to load bytecode shared object from PyPy. If all of this works, OCamlers have found a nice cheat.

pkel commented 1 year ago

Due to our build mode (native shared object) and loading with PyDLL, we are compatible with CPython only. I'll tag the wheels as cp*-abi3-*.

abi3audit does not report any errors for cp39-abi3-linux_x86_64.whl.