Python 3.11+ Compile Error: incomplete type 'PyFrameObject'

apowers313 commented 11 months ago

🐛 Bug

When compiling with Python 3.11 the following error occurs:

2023-08-09T06:14:07.8531793Z      22 | typedef struct _frame PyFrameObject;
2023-08-09T06:14:07.8532107Z         |                ^~~~~~
2023-08-09T06:14:07.8532516Z   In file included from /opt/hostedtoolcache/Python/3.11.4/x64/include/python3.11/Python.h:38,
2023-08-09T06:14:07.8533187Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/detail/common.h:124,
2023-08-09T06:14:07.8533840Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/pytypes.h:12,
2023-08-09T06:14:07.8534482Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/cast.h:13,
2023-08-09T06:14:07.8535105Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/attr.h:13,
2023-08-09T06:14:07.8535743Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/pybind11.h:45,
2023-08-09T06:14:07.8536366Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/numpy.h:12,
2023-08-09T06:14:07.8536982Z                    from /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/converter/pyconverter.cc:8:
2023-08-09T06:14:07.8537788Z   /tmp/tmpeu6rdsdd/nle-0.9.0/third_party/pybind11/include/pybind11/pybind11.h:2296:30: error: invalid use of incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
2023-08-09T06:14:07.8538503Z    2296 |             frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0));
2023-08-09T06:14:07.8538892Z         |                              ^~~~~~~~~~~~~~~~

To Reproduce (on Ubuntu 22.04)

sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt install python3.11 python3.11-dev
pip install nle (assumes installing cmake and other build instructions have already been performed)

Expected behavior

Actually compile ;)

Additional context

This occurs because Python 3.11 restructured the PyFrameObject structure (search that link for "PyFrameObject fields") which breaks old versions of pybind11. Currently the third-party directory of NLE contains a version of pybind11 that is pinned to a git submodule that is 2+ years old and won't work with Python 3.11.

cdmatters commented 11 months ago

Thanks for this and your previous issues (on WASM & gymnasium), I'll try to get to this at the same time as porting to Gymnasium. Can't promise anything on WASM though.

apowers313 commented 11 months ago

Just submitted a PR for pybind11 (I'll review the CLA later). Would you take one for Gymnasium too?

cdmatters commented 11 months ago

Would definitely review... IIRC with gymnasium there are somethings about NLE observations etc that dont fit with their API? If so, might be a breaking change

facebookresearch / nle