The Mys programming language - an attempt to create a statically typed Python-like language that produces fast binaries. See https://mys-lang.org for more information.
Following discord discussion on cross compiling, trying to canvas things to do.
reminder
build: the machine you are building on
host: the machine you are building for
target: the machine that compiler will produce binary for
What is really defining a target ?
it's an ecosystem defined by a combination of a libc+libstdc++ ( API ) , a linker (ABI) and an hardware architecture(CPU/ARCH) to run on.
What currently potential and usefull targets for mys ?
emscripten ( works , happens to be the most used platform on this earth (web) !)
wasi ( does not work yet but may be soon very interesting ).
android ( works too , happens to be the most widespread OS on this earth )
emscripten ( runs on javascript engine or wasm , all to be found in modern browsers eg chromium 81+ )
CPU/ARCH: asm.js or wasm
API : good supports for C/C++ standards and exceptions and with limited threading, no jit no native call ( browser sandboxing )
ABI: better use static linking of PIC objects.
executables are multifile : one .html or .js , a .wasm for bitcode and possibly a .data for a posix filesystem archive ( used to provide files in browsers )
wasi ( wasm / use a runtime interpreter on bytecode )
CPU/ARCH: wasm
API : no support for C setjmp, and not C++ exceptions and limited threading, can do ffi and native calls
ABI: better use static linking of PIC objects.
executables : .wasm that use a specific ABI revision from a key/value mapping ( older key called "wasi_unstable" most recent "wasi_snapshot_preview1" ) usually evolved from "cloudabi" https://github.com/NuxiNL/cloudlibc.
very close to emscripten may merge with it sooner or later.
android ( a linux mobile kernel flavour easily embedded )
CPU/ARCH: mips arm5/arm7 arm64 x86 x86_64
API: numbered from 1 to 30 with C broken locales until api 24 but overall good C++ support, https://en.wikipedia.org/wiki/Android_version_history
ABI: ELF executables using a linux-ish custom linker located at /system/bin/linker that does not respect RPATH and RTLD_ params
must be packaged in a signed ZIP file called APK.
These three arch have usually the common problem that the build cannot run the host binaries ( except maybe (quite) complex linux build + some tweak with binfmt / proot / chroot ) or that native compilation via emulation is very slow and may require admin rights.
so short answer : use cross compilation.
toolchains are provided by vendors for all those targets or usually already supported by recent versions of clang 10+
all toolchains listed here are clang versions customized for target to run on linux x86_64 systems. It may be possible to deploy those on darwin, aosp, win32 systems but it's yet out of scope.
android:
clang toolchains ( a lot of them ) grouped under the acronym NDK ( native developpement kit )
goto https://developer.android.com/ndk/downloads and pick a know-to-work version because ndk new versions break things, have silly layout and exotic build tools and TLDR licensing. cmake may work.
good to know: github actions provide ready to use NDK CI. python-for-android/beeware projects may demonstrate how to setup a ndk headless without too much user stress.
Mys ?
Mys need to know :
which toolchain is to be used
where it is located ( and how to set it up )
which api/abi/cpu
which custom restrictions/warning to apply when transpiling : maybe use a C preprocessor ? common macro are __EMSCRIPTEN__ __wasi__ __ANDROID__ __ANDROID_API__
That's too much at once and should be decomposed to avoid "black box" effect and too huge/complicated scripts unless you plan to provide some docker/vm image or unsigned prebuilts !
proposal part 1:
three root packages:
mys-cross-emsdk mys-cross-wasi mys-cross-android that can download toolchains, present and accept licenses and set compiler up
i'm strongly in favor of using a custom python sysconfig template because it would help narrow down and maybe solve lots of pip extension building problems on c-python ecosystem ( upstream step 2 )
proposal part 2:
a parent package mys-cross-compile
a generic robust parent package files which by default would target the OLDER and MOST COMPATIBLE target of each ecosystem to ensure large audience.
run real TESTSUITES on real or emulated host and offer support only on those "old stables", and maybe integrate with pip testrunner ( upstream step 3).
proposal part 3:
use small patch files to apply versionning on the parent package from a folder with a specific name scheme.
( and maybe use https://pypi.org/project/patch in case not gnu/linux for applying conversion )
When addressed by mys build/install with a --target parameter.
for eg --target=wasm_1_emscripten you would have those to apply from the matching target folder:
try to do it as much as "pip/MULTIARCH" as possible and eventually ( upstream step 1 )
Sidenotes :
All this process as been used for real ( except automated testing) with cpython/panda3d that includes a lot of third parties that mys can gain access to at least for emscripten and android via https://github.com/pmp-p/pydk
As one can see upstreaming to pip steps order is not the same as Mys logical progress that may explain why cross compilation is stalled on Python ecosystem.
Following discord discussion on cross compiling, trying to canvas things to do.
reminder
What is really defining a target ?
it's an ecosystem defined by a combination of a libc+libstdc++ ( API ) , a linker (ABI) and an hardware architecture(CPU/ARCH) to run on.
What currently potential and usefull targets for mys ?
emscripten ( runs on javascript engine or wasm , all to be found in modern browsers eg chromium 81+ )
CPU/ARCH: asm.js or wasm API : good supports for C/C++ standards and exceptions and with limited threading, no jit no native call ( browser sandboxing ) ABI: better use static linking of PIC objects. executables are multifile : one .html or .js , a .wasm for bitcode and possibly a .data for a posix filesystem archive ( used to provide files in browsers )
wasi ( wasm / use a runtime interpreter on bytecode )
CPU/ARCH: wasm API : no support for C setjmp, and not C++ exceptions and limited threading, can do ffi and native calls ABI: better use static linking of PIC objects. executables : .wasm that use a specific ABI revision from a key/value mapping ( older key called "wasi_unstable" most recent "wasi_snapshot_preview1" ) usually evolved from "cloudabi" https://github.com/NuxiNL/cloudlibc.
very close to emscripten may merge with it sooner or later.
android ( a linux mobile kernel flavour easily embedded )
CPU/ARCH: mips arm5/arm7 arm64 x86 x86_64 API: numbered from 1 to 30 with C broken locales until api 24 but overall good C++ support, https://en.wikipedia.org/wiki/Android_version_history ABI: ELF executables using a linux-ish custom linker located at /system/bin/linker that does not respect RPATH and RTLD_ params must be packaged in a signed ZIP file called APK.
more info https://pythondev.readthedocs.io/android.html signing zip/apk may require external tools or specific java versions and esoteric knowledge.
how to compile for a target ?
These three arch have usually the common problem that the build cannot run the host binaries ( except maybe (quite) complex linux build + some tweak with binfmt / proot / chroot ) or that native compilation via emulation is very slow and may require admin rights.
so short answer : use cross compilation.
toolchains are provided by vendors for all those targets or usually already supported by recent versions of clang 10+
all toolchains listed here are clang versions customized for target to run on linux x86_64 systems. It may be possible to deploy those on darwin, aosp, win32 systems but it's yet out of scope.
emscripten : clang based easy to install with
wasi: get https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-linux.tar.gz and unpack. standard clang cross compilation rules apply and work with all classic posix build tools.
android: clang toolchains ( a lot of them ) grouped under the acronym NDK ( native developpement kit ) goto https://developer.android.com/ndk/downloads and pick a know-to-work version because ndk new versions break things, have silly layout and exotic build tools and TLDR licensing. cmake may work.
good to know: github actions provide ready to use NDK CI. python-for-android/beeware projects may demonstrate how to setup a ndk headless without too much user stress.
Mys ?
Mys need to know : which toolchain is to be used where it is located ( and how to set it up ) which api/abi/cpu which custom restrictions/warning to apply when transpiling : maybe use a C preprocessor ? common macro are
__EMSCRIPTEN__ __wasi__ __ANDROID__ __ANDROID_API__That's too much at once and should be decomposed to avoid "black box" effect and too huge/complicated scripts unless you plan to provide some docker/vm image or unsigned prebuilts !
proposal part 1:
three root packages:
mys-cross-emsdk mys-cross-wasi mys-cross-androidthat can download toolchains, present and accept licenses and set compiler upproposal part 2:
a parent package
mys-cross-compilea generic robust parent package files which by default would target the OLDER and MOST COMPATIBLE target of each ecosystem to ensure large audience.proposal part 3:
use small patch files to apply versionning on the parent package from a folder with a specific name scheme. ( and maybe use https://pypi.org/project/patch in case not gnu/linux for applying conversion )
matching each target with a MULTIARCH tag inspired from python/pip and some earlied proposals on bugs.python.org (bpo) for android. https://bugs.python.org/issue?%40search_text=android+multiarch&submit=search
which CPU/ARCH to pass to the underlying C++ compiler ( clang ) which API to select which ABI to interface
that method also allow extra tag like "-testmynewapi" on simple and small patches folders to test contributors proposals.
draft for naming : https://github.com/pmp-p/pydk/issues/22
eg:
When addressed by mys build/install with a --target parameter. for eg --target=wasm_1_emscripten you would have those to apply from the matching target folder:
Sidenotes :
All this process as been used for real ( except automated testing) with cpython/panda3d that includes a lot of third parties that mys can gain access to at least for emscripten and android via https://github.com/pmp-p/pydk
As one can see upstreaming to pip steps order is not the same as Mys logical progress that may explain why cross compilation is stalled on Python ecosystem.