search

buildstream-migration / bst-staging

GNU Lesser General Public License v2.1
0 stars 0 forks source link

BuildStream doesn't hit the cache when building foreign arches #523

Open Cynical-Optimist opened 3 years ago

Cynical-Optimist commented 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @bochecha] on Jul 27, 2018, 16:39

Summary

I'm building org.freedesktop.Sdk/arm/unstable on my x86_64 laptop. The exact same Git revision has already been built in the CI.

Yet BuildStream doesn't pull from the cache, it rebuilds everything instead.

Steps to reproduce

  1. Set up qemu/binfmt on your x86_64 machine, so that building for ARM just works
  2. git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk
  3. cd freedesktop-sdk

  4. git checkout bochecha/qemu-bash

    this is the branch I'm trying to build, I won't force-push so that it remains stable, which will help test more easily than on a moving "master"

  5. cd sdk
  6. bst -o bootstrap_build_arch aarch64 -o target_arch arm build all.bst

What is the current bug behavior?

BuildStream will pull some things, and then proceed to rebuilding pretty much everything.

What is the expected correct behavior?

BuildStream should have pulled everything from the shared cache, since everything has already been built by our CI.

Relevant logs and/or screenshots

Here is an extract from the CI build logs (just a few random lines) :

fetch needed 40d8f66436c808e0d2803207cb611944d6bfa92df3e2149cf4ea8b4a6c2c8c68 bootstrap-junction.bst:coreutils.bst
     waiting 21ecc5bf231e2a2e4a8332f418729b413feb12c70d10e54df949beccff270767 bootstrap-junction.bst:diffutils.bst
fetch needed 2ada674ab2eb535bf757c75b883a73258b062c34a7eae357556cd01b257db414 bootstrap-junction.bst:files-stage1.bst
     waiting 459339a2b4c1db0948e7f3d9168f0bfd9dc06cea0f5d3a6d3929189cc6251973 bootstrap-junction.bst:zlib.bst
fetch needed 1a28b1601c89b3ffe1c106a25046b8dc66360ec48c9be47943e65c4917121fbe bootstrap-junction.bst:file.bst
     waiting f711791e4f17914f7abe0da876627d09bfca8b43f7d64195cbe4386b57a1a5e1 bootstrap-junction.bst:findutils.bst
fetch needed b1af27320610e2738c0ca6f99c71672bf8d8395003f549fe185ce63e5227384a bootstrap-junction.bst:gawk.bst
     waiting e92e604dbff0f1a6006775b3faed944c6f2a2b00d649c8b4f8c186f86e34cc64 bootstrap-junction.bst:gcc.bst

And here is an extract (the same 8 lines) from what I get on my laptop right now, running bst -o bootstrap_build_arch aarch64 -o target_arch arm build all.bst: 

     waiting 495f1f280a527df0b1b0fa2a83b887a31228bb5efc941112eb9b9150dffd8c00 bootstrap-junction.bst:coreutils.bst 
      cached ddb03abe3928bd9d7e2dc9d4e40d39ad0390aef293cebf6857be65a972c69872 bootstrap-junction.bst:diffutils.bst 
     waiting 05000dd93ec34d64b6cad910a7d12c2ed6fb2f38c9e7417b4652634cfca93794 bootstrap-junction.bst:files-stage1.bst 
      cached 887e19680fd9568f3387438e6720c0cbd2f5be08ba44c2b8023919c6b9193262 bootstrap-junction.bst:zlib.bst 
     waiting 805f803d01a8151b1bb2259482d1a25e22b8343fdb09117c8b74f5aae1648ab7 bootstrap-junction.bst:file.bst 
     waiting 375a0032b05c89619f022783960f6def87597de540a38f03fd52927d8421c814 bootstrap-junction.bst:findutils.bst 
     waiting b9f24859a032a4faaada15c55762feed3e644caeea23910713911633fdaab2b3 bootstrap-junction.bst:gawk.bst 
     waiting a77694f1bd40ada7fbf416dac1597aa49c1db65339081d1770798dd3fae140d4 bootstrap-junction.bst:gcc.bst

As you can see, the cache keys are different.

I can't figure out what difference between my laptop and the CI could cause this though. They have the same project.conf and buildstream.conf, the same version of bst and bst-external.

The only difference seems to be the arch of the host, but I don't think that should be considered when computing the cache key.

[Gitlab user @jjardon]

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @valentindavid] on Jul 31, 2018, 16:27

It seems SandboxConfig uses os.uname to add host architecture to the cache keys.

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @bochecha] on Sep 13, 2018, 09:23

It seems SandboxConfig uses os.uname to add host architecture to the cache keys.

I tried hardcoding that to 'aarch64', but bst still didn't hit the cache.

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @tristanvb] on Sep 13, 2018, 14:12

[Gitlab user @bochecha] it is indeed confusing to me that hard coding that did not produce the same key.

Do you think this might be a symptom of #641 ? I.e. max-jobs variable being used directly in build instructions somewhere ?

That said, the comment in the referred code of SandboxConfig is pretty telling. BuildStream does not support cross building, at least not yet.

It is intended that to support that, we need:

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @bochecha] on Sep 14, 2018, 07:22

changed the description

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @bochecha] on Sep 28, 2018, 09:42

I tried again with Buildstream 1.2.2.

Do you think this might be a symptom of #641 ? I.e. max-jobs variable being used directly in build instructions somewhere ?

That could have been the case in the past, but we don't use max-jobs any more thanks to a contribution by [Gitlab user @tacgomes] (https://gitlab.com/freedesktop-sdk/freedesktop-sdk/merge_requests/513)

The latest FDO Sdk 18.08 uses max-jobs in exactly one place:

environment-nocache:
  - MAXJOBS

environment:
  MAXJOBS: "%{max-jobs}"

It shouldn't cause any problem, since it's an environment-nocache variable?

In any case, the native Aarch64 build in the CI had the following key for all.bst:

On my x86_64 machine, if I run bst -o bootstrap_build_arch aarch64 -o target_arch aarch64 show all.bst I get:

If I modify buildstream/sandbox/_config.py, hardcoding 'aarch64' as the arch, I get:

So hardcoding the arch in that file does have an impact on the cache key.

But there must be something else that also does, somewhere?

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @abderrahimk] on Oct 13, 2018, 20:18

[Gitlab user @bochecha]: This is weird. Hardcoding works for me. (using freedesktop-sdk/freedesktop-sdk[Gitlab user @e6939dcbaa31058434ca5b025db23733d11b05af], I get 364ee5a998a83e70d988d32e2dbd3220366214ae332e9c37ea55890cd93f0c40, which is the same as the CI). Are you sure you are modifying the bst-1.2 branch?

Another case I'd like to use this for is setting a 32-bit personality (using setarch) to build i686 on x86_64 without the buildscripts thinking they are cross compiling.

I did some tests, and here is what I get (on my x86_64). I ran uname -m with different architectures for the uname binary:

So the thing is we cannot know for sure what running uname -m in the sandbox will give before we actually run it.

It is intended that to support that, we need:

  • SandboxConfig to ask for the correct machine architecture
  • BuildStream to find out if there is a sandbox implementation available at runtime for the given architecture; and to either use that implementation or bail out with an error.

So IIUC, we need an option to pass the sandbox architecture to buildstream, and this will be used instead of os.uname() (in both the cache key, and as a default for options of type arch).

Then, we need to check whether we can run the uname binary in the sandbox, and whether it returns the correct value.

And to support my additional requirement, we also need to call bwrap while setting a personality using setarch.

Does this sound good?

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @abderrahimk] on Nov 4, 2018, 09:08

Today, I found this nifty tool https://github.com/kilobyte/arch-test. It can tell whether your machine can run binaries of the given arch (or list all the arches your machine can run binaries of).

The problem is that it gives architectures as Debian names them (amd64 instead of x86_64 for example), but I think this shouldn't be a problem.

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @juergbi] on Nov 4, 2018, 09:35

So IIUC, we need an option to pass the sandbox architecture to buildstream, and this will be used instead of os.uname() (in both the cache key, and as a default for options of type arch).

Yes, see https://mail.gnome.org/archives/buildstream-list/2018-November/msg00003.html for my proposal how to configure this.

And to support my additional requirement, we also need to call bwrap while setting a personality using setarch.

Agreed, we should set the personality as well.

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @valentindavid] on Nov 22, 2018, 11:36

marked this issue as related to #775

Cynical-Optimist commented 3 years ago

In GitLab by [Gitlab user @abderrahimk] on May 16, 2020, 19:13

mentioned in merge request !1934