CPython integration help needed

linuxboot / heads

A minimal Linux that runs as a coreboot or LinuxBoot ROM payload to provide a secure, flexible boot environment for laptops, workstations and servers.

https://osresearch.net/

GNU General Public License v2.0

1.42k stars 185 forks source link

CPython integration help needed #689

Open tlaurion opened 4 years ago

tlaurion commented 4 years ago

Micropython would not fit the bill of wyng dependencies in the long run.

CPython could work with additional space freed with modified IFD and Neutered ME, but I would need to understand how to properly cross-compile it.

Outcome would be chipsec inclusion, wyng-backups inclusion permitting to have factory disk restore/integrity validation inside of Heads.

Linked #233

osresearch commented 4 years ago

I'm using micropython in other projects with good success, but those embedded systems don't have deep dependency trees.

I'll poke at cpython to see if there is an easy way to apply the fix from 360651

osresearch commented 4 years ago

I first build a native version for the build machine and then build a cross compiled version using the native as the $PYTHON_FOR_BUILD that appears to work in a chroot environment:

CC=$HEADS/crossgcc/bin/x86_64-linux-musl-gcc \
PYTHON_FOR_BUILD=`pwd`/../build/python \
ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no \
have_openssl=no \
../configure \
  --build x86_64-linux-musl \
  --host x86_64-linux-gnu \
  --disable-ipv6 \
  --prefix=/tmp/root/usr/local \

make "passes", although many of the extensions fail. It also appears that quite a bit of the code ends up ignoring $(CC) and building with gcc instead (all of the cpython/Modules directory?).

Another problem is that python wants openssl, which is a massive dependency.

osresearch commented 4 years ago

Yeah, the wrong compiler is definitely going to make this a mess:

~ # python
Python 3.9.0a4+ (heads/master-dirty:efc28bb, Mar  5 2020, 19:38:39) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: Error loading shared library libffi.so.6: No such file or directory (needed by /usr/local/lib/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so)
>>>

osresearch commented 4 years ago

Oops, I swapped the --host and --build values. Rebuilding with this for the configuration:

CC=~/heads/install/bin/musl-gcc \
 CXX=~/heads/install/bin/musl-gcc \
READELF=~/heads/crossgcc/x86_64-linux-musl/x86_64-linux-musl/bin/readelf \
PYTHON_FOR_BUILD=`pwd`/../build/python \
ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no \
have_openssl=no \
../configure \
  --host x86_64-linux-musl \
  --build x86_64-linux-gnu \
  --disable-ipv6  \
  --prefix=/tmp/root/usr/local \

Takes forever due to warnings in Modules/unicodename_db.h, but does appear to use the correct gcc this time. However, import ctypes produces the same error, so something is still borked.

tlaurion commented 4 years ago

@osresearch : Anything practical to provide here to help fix things upstream?

osresearch commented 4 years ago

I was idly wondering about building a fakeroot environment so that the build environment is "native" and doesn't have the cross compilation issues. This would mean building a bootstrap compiler to build the musl native compiler, although since we have busybox already built for musl we have most of the rest of the environment that we would need.

tlaurion commented 4 years ago

@osresearch : ping?