Compiling Python

[ahgamut]

https://github.com/ahgamut/python27
https://github.com/ahgamut/cpython/tree/cosmo_py27

The assert macro needs to be changed in cosmopolitan.h to enable compilation (see #138). Afterwards, just clone the repo and run superconfigure.

Python 2.7.18 compiled seamlessly once I figured out how autoconf worked, and what flags were being fed to the source files when running make. I'm pretty sure we can compile any C-based extensions into python.exe -- they just need to compiled/linked with Cosmopolitan, with necessary glue code added to the Python source. For example, I was able to compile SQLite into python.exe to enable the internal _sqlite module.

The compiled APE is about 4.1MB with MODE=tiny (without any of the standard modules, the interpreter alone is around 1.6MB). Most of the modules in the stdlib compile without error. The _socketmodule (required for Python's simple HTTP server) doesn't compile, as it requires the structs from netdb.h.

On Windows, the APE exits immediately because the intertpreter is unable to find the platform-specific files. Module/getpath.c and Lib/site.py in the Python source try to use absolute paths from the prefixes provided during compilation; Editing those files to search the right locations (possibly with some zipos magic) ought to fix this.

[ahgamut]

The superconfigure build is the single executable I was referring to -- I add the packages/app I need into the CPython source tree.

For example: https://github.com/ahgamut/superconfigure/releases/download/z0.0.27/datasette.zip contains a single-file python executable that can run the datasette app.

If I want to add more pure-python libraries to the above executable, I can do it using the zip tool as follows:

mv datasette datasette.com
unzip -vl datasette.com
./datasette.com -m pip download tqdm # sample pure-python library
mkdir -p Lib/site-packages
unzip tqdm*.whl -d ./Lib/site-packages
zip -qr ./datasette.com Lib/site-packages
# now tqdm is part of the python executable
mv datasette.com datasette
rm -rf ./Lib/
./datasette -c 'import tqdm'

related screenshot using above executable from superconfigure, following the shell scripts:

So the datasette binary from superconfigure is a single-file python executable, that works across (Linux/MacOS/BSDs/Windows on x86_64 and Linux/MacOS on aarch64), and I can add pure-python libraries to my executable using pip and the zip command when I need to. This covers my use cases for now. C extensions can be done as well (like I have done for PyYAML and markupsafe), but I think that part of the build can be revamped after I test ctypes behavior with the latest cosmo update.

[EirikJaccheri]

Hi again, I sucessfully managed to install zstandard following your recipe. But i am struggling to install lz4.

what i have done:

1. Created the folder superconfigure/compress/lz4-1.9.4

2. created a superconfigure file (attached screenshot)

As you can see i commented out the./configure part of the script since there is no configure script when i untar lz4-1.9.4.tar.gz.

When i source source vars/x86_64 and run the superconfigure script i get the following error message:

Cleaning completed compiling static library compiling dynamic library 1.9.4 x86_64-unknown-cosmo-cc: -shared not supported make[1]: [Makefile:122: liblz4.so.1.9.4] Error 1 make[1]: Waiting for unfinished jobs.... make: *** [Makefile:57: lib-release] Error 2

Can you point me to what i am doing wrong? @ahgamut

Also: if i get the compress/lz3-1.9.4/superconfigure script to work, is the only remaining step to unzip the .whl file in Lib/site-packages and run .github/scripts/build/datasette?

Thanks for your help, Eirik

[ahgamut]

@EirikJaccheri I added a build script for lz4 on superconfigure based on your script above: https://github.com/ahgamut/superconfigure/commit/325438d6f1694f0b24518c8e894f67f4fcad51a6

Turns out we have to patch the Makefiles on lz4 a little bit, to avoid building a shared object.

I sucessfully managed to install zstandard following your recipe. But i am struggling to install lz4.

you mean you managed to build the zstandard python package as part of the CPython build? Nice! With the above lz4 build, you just need to add the lz4 python package similar to how you added zstandard.

[EirikJaccheri]

I managed to install lz4 and now clickhouse_connect is working! It would be great if we could add these repositories to the repository. However i am not sure how to submit the pull request.

The way i added the file was by manually adding files to python/cpy311-datasette/datasette/Modules/, python/cpy311-datasette/datasette/Modules/ and changing Modules/Setup. (as you can see inn my fork of datasette: https://github.com/EirikJaccheri/cpython/tree/datasette)

How can i make it so that the packages are created from the python/cpy311-datasette/superconfigure script?

Also, how do i clean the superconfigure repo?

Thanks for all your help, Eirik

[ahgamut]

I managed to install lz4 and now clickhouse_connect is working!

Nice! can you post a screenshot of the working clickhouse_connect?

It would be great if we could add these repositories to the repository. However i am not sure how to submit the pull request.

I am not sure what clickhouse_connect does, tbh. If it is https://github.com/ClickHouse/clickhouse-connect, here's what we can do: I will update the datasette build or the pypack1 build to include it, probably as part of the next superconfigure release? I'll use your fork of cpython as reference.

Also, how do i clean the superconfigure repo?

you can clean the superconfigure repo via make clean or git clean -f -d -x.

[EirikJaccheri]

That is indeed the correct library. Here is a screenshot of me importing clickhouse_connect:

Great that you will include it in the next release!

I work in a research institution and we are planing to use clickhouse connect to upload data from experiments to a centralized server. The ./datasette.com excecutable removes the barrier to entry of installing python with the correct packages, especially from windows:-)

[ahgamut]

https://github.com/ahgamut/superconfigure/releases/download/z0.0.28/pypack1.zip The above executable has the clickhouse_connect pure-python library.