Incompatibility with Yocto's Pre-Built buildtools Tarball

bantu commented 3 years ago

Yocto Context

https://www.yoctoproject.org/docs/3.1.3/mega-manual/mega-manual.html#required-git-tar-python-and-gcc-versions describes how to install a tarball containing the minimal build tools (git, tar, python, gcc) to bootstrap a full yocto build using bitbake. Installing without root permissions is explicitly supported and seems to work pretty well from my experience. As described in the manual, the binaries from the buildtools tarball can be added to PATH by sourcing the provided environment file manually or for example by sourcing it from .profile.

Setup Context

My setup is this: I am provided an unprivileged user on a Linux system. I can do whatever I want in this user's home directory. I explicitly do not want to make any assumptions about the build tools provided by the host system and want to use Yocto's Pre-Built buildtools tarball to bootstrap the build system. This seems to be inline with the goals of kas and the goal of reproducible builds. Afterall, my build environment is unaffected by changes made to the host system.

I use the python version provided by the tarball to setup a python venv in which I install kas via pip
kas requires the git binary provided by the tarball to clone repositories

Problem

kas resets the PATH variable to /usr/sbin:/usr/bin:/sbin:/bin in https://github.com/siemens/kas/blob/2.2/kas/libkas.py#L231. This goes against my goals of using the pre-built buildtools tarball to achieve reproducible builds. Indeed, this results in kas attempting to use only system-provided tools and failing if some tools are missing on the system (although I explicitly installed them as per the Yocto manual and everything works fine without kas).

Related Reports

There is #27 which describes a similar problem and related discussion at https://groups.google.com/g/kas-devel/c/JXD7wQLu6Cg/m/u0q1uBDVBgAJ. The root cause of this issue is the same, however, my issue concerns setting up the dependencies for kas and bitbake in the first place.

jan-kiszka commented 3 years ago

There are two ways to model that today:

provide the build environment via a container, like we do via ghcr.io/siemens/kas/kas:<version> and kas-container
add an own env: section to the kas file that specifies the deviations from the standard env

bantu commented 3 years ago

Thanks for the speedy reply, Jan.

Neither of the two options look decent to me as of right now.

Containers add a whole stack of new dependencies and introduce various other issues. One large issue I see with containers is that container images being depended on have to be audited (for backdoors etc).

env: looks okay on first thought, but turns out to be rather inappropriate on closer inspection. Yes, I can add /home/gitlab-runner/buildtools/bin to PATH via the kas file, but if I commit this into the repository, it will immediately break builds for /home/hans and /home/franz. This breakage is introduced by kas and does not seem to exist with only bitbake. To avoid this problem, I can add a script which runs before kas and adjusts the kas env such that PATH is setup as expected. However, this is a rather fragile solution and silly for sure.

How about adding a command line option allowing PATH to be preserved?

jan-kiszka commented 3 years ago

Thanks for the speedy reply, Jan.

Neither of the two options look decent to me as of right now.

Containers add a whole stack of new dependencies and introduce various other issues. One large issue I see with containers is that container images being depended on have to be audited (for backdoors etc).

In contrast to the buildtools tarball, containers are self-contained, only requiring a compatible kernel underneath. That tarball still needs a host distro and, thus, has additional userland dependencies (even if they are widely agnostic to the specific version or flavor). And if you audit them or the Debian-based container kas puts together, doesn't matter a lot.

There is a reason why we use full chroots (i.e. containers in the modern word) in order to permit smooth transition between arbitrary developer workstations (including WSL2) and CI environments (being container based today).

env: looks okay on first thought, but turns out to be rather inappropriate on closer inspection. Yes, I can add /home/gitlab-runner/buildtools/bin to PATH via the kas file, but if I commit this into the repository, it will immediately break builds for /home/hans and /home/franz. This breakage is introduced by kas and does not seem to exist with only bitbake. To avoid this problem, I can add a script which runs before kas and adjusts the kas env such that PATH is setup as expected. However, this is a rather fragile solution and silly for sure.

You do not need to specify a default PATH. Listing PATH in env will mean that this is taken from your host env. However...

How about adding a command line option allowing PATH to be preserved?

...opening up PATH is generally a bad idea. It allows for ~/bin or /opt/random-stuff being available on one host, allowing the build to work, while it might be missing on another. That's why this can't become a default in kas.

That said, I'm not opposing completely to simplify the usage of buildtool tarballs, specifically if that is a common pattern. But then we need to define how to cleanly and deterministically switch to them, and only them. Feel free to follow up on that on the ML.

jpuhlman commented 3 years ago

I literally was just looking at this recently. The following patch would do it. https://github.com/jpuhlman/kas/commit/2207a9074c85f3e9be11ce1ca5ddee4cf321aed4

It doesn't open up the PATH to everything, it just looks for the build tools environment file and adds it to the pre-enviornment script before executing the build.

bantu commented 3 years ago

You do not need to specify a default PATH. Listing PATH in env will mean that this is taken from your host env.

How would I go about doing that? The kas documentation says that env is of type dict. I tried

[...]
env:
  PATH: "PATH1:PATH2:PATH3"

but this fails with

2020-11-18 12:30:10 - ERROR    - sequence item 0: expected str instance, NoneType found
Traceback (most recent call last):
  File "/home/[...]/kas-venv/lib/python3.8/site-packages/kas/kas.py", line 162, in main
    sys.exit(kas(sys.argv[1:]))
  File "/home/[...]/kas-venv/lib/python3.8/site-packages/kas/kas.py", line 150, in kas
    if plugin().run(args):
  File "/home/[...]/kas-venv/lib/python3.8/site-packages/kas/build.py", line 110, in run
    macro.run(ctx, args.skip)
  File "/home/[...]/kas-venv/lib/python3.8/site-packages/kas/libcmds.py", line 63, in run
    command.execute(ctx)
  File "/home/[...]/kas-venv/lib/python3.8/site-packages/kas/build.py", line 136, in execute
    logging.info('%s$ %s', ctx.build_dir, ' '.join(cmd))
TypeError: sequence item 0: expected str instance, NoneType found

(using kas version 2.1.1)

jan-kiszka commented 3 years ago

I literally was just looking at this recently. The following patch would do it. jpuhlman@2207a90

It doesn't open up the PATH to everything, it just looks for the build tools environment file and adds it to the pre-enviornment script before executing the build.

That is a pattern I could imagine for upstream. If you consider it done, please submit as patch, @jpuhlman.

jan-kiszka commented 2 years ago

What happened to this?

alexandrebelloni commented 1 year ago

I just sent an updated patch for this issue. We have multiple hosts were we want to use buildtools instead of the host tools from the distribution.

henning-schild commented 1 year ago

I will not say a word and "Unsubscribe", we have a mailing list for that stuff and issues are strongly discouraged.

siemens / kas