riscv-collab / riscv-gnu-toolchain

GNU toolchain for RISC-V, including GCC
Other
3.56k stars 1.17k forks source link

Error: found no usable tomli #1615

Open karaketir16 opened 1 week ago

karaketir16 commented 1 week ago

Error while building with build-sim SIM=qemu in Ubuntu 22.04

 => => # python version: Python 3.10.12
 => => # mkvenv: Creating non-isolated virtual environment at 'pyvenv'
 => => # *** Ouch! ***
 => => # found no usable tomli, please install it
 => => # make: *** [Makefile:1124: stamps/build-qemu] Error 1

Dockerfile to reproduce

# Start with Ubuntu 22.04 as the base image
FROM ubuntu:22.04

# Set environment variables to avoid interactive prompts during build
ENV DEBIAN_FRONTEND=noninteractive

# Update and install required dependencies
RUN apt-get update && apt-get install -y \
    autoconf \
    automake \
    autotools-dev \
    curl \
    python3 \
    python3-pip \
    libmpc-dev \
    libmpfr-dev \
    libgmp-dev \
    gawk \
    build-essential \
    bison \
    flex \
    texinfo \
    gperf \
    libtool \
    patchutils \
    bc \
    zlib1g-dev \
    libexpat-dev \
    ninja-build \
    git \
    cmake \
    libglib2.0-dev \
    && rm -rf /var/lib/apt/lists/*  # Clean up apt cache to reduce image size

# Clone the riscv-gnu-toolchain repository
RUN git clone https://github.com/riscv/riscv-gnu-toolchain /riscv-gnu-toolchain

# Build the toolchain
WORKDIR /riscv-gnu-toolchain
RUN ./configure --prefix=/opt/riscv --enable-multilib && \
    make -j$(nproc) newlib linux build-sim SIM=qemu && \
    make install && \
    rm -rf /riscv-gnu-toolchain # Clean up to reduce image size

# Set the PATH environment variable to include the RISC-V toolchain
ENV PATH=/opt/riscv/bin:$PATH

# Set the working directory (optional)
WORKDIR /workspace

# Set the default command to run (you can override this when running the container)
CMD ["bash"]
mickflemm commented 4 days ago

I couldn't reproduce this, QEMU's build script creates a python venv and installs needed packages inside, this is the output in my case on a clean ubuntu 22.04 container (instantiated using lxc -n ubuntu-test -t download):

Using './build' as the directory for build output
python determined to be '/usr/bin/python3'
python version: Python 3.10.12
mkvenv: Creating non-isolated virtual environment at 'pyvenv'
mkvenv: checking for tomli>=1.2.0
mkvenv: installing tomli>=1.2.0
mkvenv: checking for meson>=0.63.0
mkvenv: installing meson==1.2.3
The Meson build system
Version: 1.2.3
Source dir: /yarvt/sources/riscv-qemu
Build dir: /yarvt/sources/riscv-qemu/build
Build type: native build
Project name: qemu
Project version: 8.2.7
C compiler for the host machine: cc -m64 -mcx16 (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc -m64 -mcx16 ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64

As you can see it installed tomli inside the virtual environment, so there is no need to install it on the host OS as well. Maybe there it can't install the package in your case, network issues ?

karaketir16 commented 4 days ago

I will try again with the same Dockerfile. It might have been a network issue, as you suggested.

I'll let you know.

karaketir16 commented 4 days ago

It happened again, and it's on GitHub Actions. It's probably not a network issue on their side. Here's the link to the logs: https://github.com/karaketir16/test-docker/actions/runs/11913130748/job/33198181961#step:3:12935.

#8 688.8 checking that generated files are newer than configure... done
#8 688.8 configure: creating ./config.status
#8 688.8 mkvenv: Creating non-isolated virtual environment at 'pyvenv'
#8 688.8 config.status: creating Makefile
#8 688.9 mkdir -p -- ./libbacktrace
#8 688.9 Configuring in ./libbacktrace
#8 689.1 
#8 689.1 *** Ouch! ***
#8 689.1 
#8 689.1 found no usable tomli, please install it 
#8 689.1 
#8 689.1 
#8 689.1 make: *** [Makefile:1124: stamps/build-qemu] Error 1
#8 689.1 make: *** Waiting for unfinished jobs....
#8 689.1 mkdir -p -- ./libdecnumber
karaketir16 commented 4 days ago

QEMU's build script creates a python venv and installs needed packages inside,

That's exactly where it fails—it cannot create the virtual environment (venv) https://gitlab.com/qemu-project/qemu/-/blob/master/python/scripts/mkvenv.py?ref_type=heads#L719

mickflemm commented 4 days ago

Ah now I managed to reproduce this...

https://wiki.qemu.org/ChangeLog/8.2 : "Building QEMU now uses the tomli library if Python is older than version 3.11. However, version 2.0.1 is bundled in case tomli is not installed on the host."

https://wiki.qemu.org/ChangeLog/9.1 : "When using Python 3.10 or older, building QEMU requires the tomli package to be installed on the host. (The dependency was introduced in QEMU 8.2 but until now QEMU included a vendored copy of the library)."

In our case we only run make_report (which compiles QEMU etc) on ubuntu 24.04 runners that come with python 3.12 (so it doesn't require tomli). However your dockerfile is for ubuntu 22.04 that has python 3.10, in my setup (with the ubuntu 22.04 container) I noticed I was still building 8.2, that's why it worked fine.

So if someone does make_report on ubuntu 22.04 (which is disabled in our CI) they'll need to install tomli. I guess it makes sense to add this to the README, it should be true for other distros that use python < 3.11 as well.

karaketir16 commented 4 days ago

I checked the following Docker images:

I mean "has" to indicate the default Python3 version available when installed.

I couldn’t find Docker images for other distributions.

Installing python3-tomli on ubuntu:24.04 seems to have no side effects. The script first checks for import tomllib, so #1614 is valid for both Ubuntu versions.

Here’s a sample script for clarity:

HAVE_TOMLLIB = True
try:
    import tomllib
except ImportError:
    try:
        import tomli as tomllib
    except ImportError:
        HAVE_TOMLLIB = False

if not HAVE_TOMLLIB:
    raise Exception("Neither tomllib nor tomli found.")
else:
    print("Tomli or tomllib found. No problem.")
TommyMurphyTM1234 commented 1 day ago

Can somebody clarify what, if anything, needs to be done to address and close this issue? E.g.

  1. Nothing? 🙂
  2. Modify setup-apt.sh?
  3. Update README.md?
  4. Something else?
karaketir16 commented 1 day ago

Can somebody clarify what, if anything, needs to be done to address and close this

  1. Nothing? 🙂
  2. Modify setup-apt.sh?
  3. Update README.md?
  4. Something else?
  1. If this file is just for github-actions, no, no need to modify it. But if it's an example of requirements, it would be better to add.
  2. is needs to be done. #1614