search

robotpy / roborio-python

CPython ipkg builds for RoboRIO
0 stars 0 forks source link

Add LTO build #2

Open virtuald opened 1 month ago

virtuald commented 1 month ago

LTO is faster, seems nice? Unfortunately, I tried and I can't build it on the RoboRIO directly... so, will need to cross-compile it on the host, which will require a three part build to do LTO + profile guided optimizations:

I tried it and failed (python failed to initialize for some reason), will need to try later in the beta period (but you can try too if you're awesome at building things):


opkg install rsync

docker run --rm -it \
    -v $(pwd):/media/sda1/Python \
    -w /media/sda1/Python \
    --hostname robotpy-builder \
    wpilib/roborio-cross-ubuntu:2024-22.04

tar -xf Python-3.13.0.tgz
cd Python-3.13.0

configure ..

note: local build has bz2, lzma

.. TODO: change unpack.py to fix the .pc files to be relocatable

remote: make profile-run-stamp .. no
    patch it
    run it
    rsync it back
    - might need to mess with timestamps

make profile-gen-stamp
rsync -av Python-3.13.0 roborio:/media/sda1/Python-3.13.0
    LD_LIBRARY_PATH=$(pwd) ./python -m test --pgo

make clean-retain-profile
touch profile-run-stamp

make profile-opt
make DESTDIR=/media/sda1/install install

TODO download.sh:

#!/bin/bash

wget http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libffi6_3.2.1-r0.535_cortexa9-vfpv3.ipk
wget http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libffi-dev_3.2.1-r0.535_cortexa9-vfpv3.ipk

wget http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libexpat-dev_2.2.5-r0.144_cortexa9-vfpv3.ipk
wget http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libsqlite3-dev_3.22.0-r0.145_cortexa9-vfpv3.ipk

wget http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libz1_1.2.11-r0.207_cortexa9-vfpv3.ipk
wget http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libz-dev_1.2.11-r0.207_cortexa9-vfpv3.ipk

# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libexpat-dev
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libexpat-dev_2.2.5-r0.144_cortexa9-vfpv3.ipk.
# Downloaded libexpat-dev as libexpat-dev_2.2.5-r0.144_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libffi6
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libffi6_3.2.1-r0.535_cortexa9-vfpv3.ipk.
# Downloaded libffi6 as libffi6_3.2.1-r0.535_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libffi6-dev
#  * opkg_download_cmd: Cannot find package libffi6-dev.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libffi-dev
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libffi-dev_3.2.1-r0.535_cortexa9-vfpv3.ipk.
# Downloaded libffi-dev as libffi-dev_3.2.1-r0.535_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libsqlite-dev
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/extra/cortexa9-vfpv3/libsqlite-dev_2.8.17-r7.51_cortexa9-vfpv3.ipk.
# Downloaded libsqlite-dev as libsqlite-dev_2.8.17-r7.51_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libsqlite
#  * opkg_download_cmd: Cannot find package libsqlite.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libsqlite3
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libsqlite3-0_3.22.0-r0.145_cortexa9-vfpv3.ipk.
# Downloaded libsqlite3-0 as libsqlite3-0_3.22.0-r0.145_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download liblzma6
#  * opkg_download_cmd: Cannot find package liblzma6.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download liblzma5
#  * get_pkg_url: Package liblzma5 is not available from any configured src.
# Downloaded liblzma5 as (null).
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download liblzma-dev
#  * opkg_download_cmd: Cannot find package liblzma-dev.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libbz2-dev
#  * opkg_download_cmd: Cannot find package libbz2-dev.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download bzip2-dev
#  * get_pkg_url: Package bzip2-dev is not available from any configured src.
# Downloaded bzip2-dev as (null).
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libbz2-1
#  * get_pkg_url: Package libbz2-1 is not available from any configured src.
# Downloaded libbz2-1 as (null).
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libffi6
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libffi6_3.2.1-r0.535_cortexa9-vfpv3.ipk.
# Downloaded libffi6 as libffi6_3.2.1-r0.535_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download# ls
# libexpat-dev_2.2.5-r0.144_cortexa9-vfpv3.ipk   libsqlite-dev_2.8.17-r7.51_cortexa9-vfpv3.ipk
# libffi-dev_3.2.1-r0.535_cortexa9-vfpv3.ipk     libsqlite3-0_3.22.0-r0.145_cortexa9-vfpv3.ipk
# libffi6_3.2.1-r0.535_cortexa9-vfpv3.ipk
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libexpat
#  * opkg_download_cmd: Cannot find package libexpat.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libexpat2
#  * opkg_download_cmd: Cannot find package libexpat2.
# admin@roboRIO-2423-FRC:/media/sda1/download# opkg download libexpat1
# Downloading http://download.ni.com/ni-linux-rt/feeds/academic/2023/arm/main/cortexa9-vfpv3/libexpat1_2.2.5-r0.144_cortexa9-vfpv3.ipk.
# Downloaded libexpat1 as libexpat1_2.2.5-r0.144_cortexa9-vfpv3.ipk.
# admin@roboRIO-2423-FRC:/media/sda1/download#

unpack.py for unpacking ipk:

#!/usr/bin/env python3

import argparse
import pathlib
import subprocess
import tempfile
import sys

def subprocess_must_run(*args, **kwargs):
    """Run a subprocess verbosely and exit if there is an error"""
    try:
        print("+", *args[0])
        subprocess.run(check=True, *args, **kwargs)
    except subprocess.CalledProcessError as cbe:
        print(cbe, file=sys.stderr)
        sys.exit(cbe.returncode)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("infile", type=pathlib.Path)
    parser.add_argument("outdir", type=pathlib.Path)
    args = parser.parse_args()

    infile = args.infile.absolute()
    outfile = args.outdir.absolute()

    with tempfile.TemporaryDirectory() as d:
        subprocess_must_run(["ar", "x", str(infile)], cwd=d)
        subprocess_must_run(["tar", "-xf", "data.tar.xz", "-C", str(outfile)], cwd=d)
virtuald commented 1 month ago

It was suggested that using BOLT would also be useful. https://github.com/python/cpython/pull/95908