bst build --track-all doesn't build anything #1014

Open Cynical-Optimist opened 4 years ago

See original issue on GitLab In GitLab by [Gitlab user @abderrahimk] on May 2, 2019, 07:22

Summary

Sometimes, running bst build with --track-all behaves as if I ran bst track. It tracks all element but doesn't try to build or pull anything.

Note

This issue is mostly fixed, but it seems to reproduce spuriously now, keeping this open to record any information pertaining to any further sightings of this issue.

Steps to reproduce

Using branch abderrahim/gnome-boot of gnome-build-meta, run

bst --no-strict --on-error continue build --track-all vm/desktop-vm-image-x86_64.bst

What is the current bug behavior?

Pipeline Summary
  Total:       613
  Session:     555
  Track Queue: processed 232, skipped 13, failed 0
  Pull Queue:  processed 0,   skipped 0,  failed 0
  Fetch Queue: processed 0,   skipped 0,  failed 0
  Build Queue: processed 0,   skipped 0,  failed 0
  Push Queue:  processed 0,   skipped 0,  failed 0

What is the expected correct behavior?

bst should try to pull/build things.

Other relevant information

This is using bst 1.2.6, in non strict mode.

BuildStream version affected: /milestone %BuildStream_v1.2

In GitLab by [Gitlab user @tristanvb] on May 2, 2019, 11:41

Oh my... time for more regression tests...

In GitLab by [Gitlab user @abderrahimk] on May 2, 2019, 11:43

btw, I forgot to mention this happened before, and I thought !1286 would fix it. But apparently not.

In GitLab by [Gitlab user @tristanvb] on May 3, 2019, 06:19

Marking critical since this is a very basic essential feature that is simply not working at all.

In GitLab by [Gitlab user @tristanvb] on May 7, 2019, 08:29

I've reproduced. Actually I'm having errors timing out while trying to track a few elements, but it shoudn't change anything, we should still be pulling and building but we seem not to be.

My results were:

Pipeline Summary
  Total:       613
  Session:     613
  Track Queue: processed 230, skipped 13, failed 3 
  Pull Queue:  processed 0,   skipped 0,  failed 0 
  Fetch Queue: processed 0,   skipped 0,  failed 0 
  Build Queue: processed 0,   skipped 0,  failed 0

I'm having a hard time reproducing this in a test case though; I've adjusted tests/frontend/buildtrack.py to add a --track-all case, and added assertions that things actually get built in test_build_track(), those tests seem to pass fine, need to figure out what is different in the gnome-build-meta case.

In GitLab by [Gitlab user @tristanvb] on May 7, 2019, 09:45

Have to leave this for today, my findings so far are that:

This seems not to be related to dependency types

Modifying the test_build_track() case such that we run the tests with dependencies as runtime only, build only, or both, does not cause this issue to arise.
This seems not to be related to having pull queues active

Building your same branch with artifact servers disabled for both GNOME and freedesktop-sdk projects, causing no pull queue to be used in the scheduler, does not cause this issue to arise.

I will take another look tomorrow and perhaps start sprinkling in some trace logging statements, any other ideas of how this build scenario differs from the test cases in question is welcome.

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 06:58

I've been debugging this more today, and I found that currently I actually cannot reproduce the issue in the same way that [Gitlab user @abderrahimk] was able to.

Contrary to the initial report which shows that all elements were successfully tracked, when I run the test the following elements fail to be tracked, as they appear to refer to upstream sources which do not exist:

core-deps/liboauth.bst
core-deps/espeak.bst
core-deps/libwacom.bst

As [Gitlab user @abderrahimk] has pointed out, libwacom has moved to github.

I initially thought this was insignificant, and that we should see elements being built regardless of the failing 3 elements, but this is not true - so far the behavior I have been witnessing is correct behavior.

The logic by which the behavior I am seeing is correct is as follows:

BuildStream does not try to build elements which can instead be downloaded
- This is specifically significant for an element like desktop-vm-image.bst, as the x86image has build only dependencies, and also depends on a compose element which of course also has build only dependencies
- We make an effort to avoid fetching and building of elements found in build only dependency branches, if we can download the build-only dependency itself from the cache, then there is no need to process its dependencies
BuildStream however needs to determine the cache key of elements it intends to attempt to download
If an element is selected for tracking, it's cache key, and the cache keys of it's reverse dependencies must remain undetermined until tracking completes
The fact that 3 elements are unable to be tracked, causes the entire branch of build-only dependencies to never be processed.

We do not end up scheduling the processing of these elements found only in build-dependency branches of the graph until we have at least determined the cache key of the first build-only dependency, as that might result in our ability to download that element instead of processing the rest of the graph.

That said, this is only the behavior I have observed, I will try to reproduce this in a scenario where the tracking of build-only dependencies does not fail.

Furthermore, I wonder about my observation yesterday:

Building your same branch with artifact servers disabled for both GNOME and freedesktop-sdk projects, causing no pull queue to be used in the scheduler, does not cause this issue to arise.

Now that I've observed what is blocking elements from being processed, this leads me to question whether the logic which blocks elements from being processed until we know that we cannot download them from a remote, should not be active when there are no remotes configured.

If we know that there are no remotes where artifacts can be downloaded from, we could start processing elements more quickly, without blocking on the full build dependency branches to be tracked before fetching an building commences.

I think that this is a distinctly separate issue, though.

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 07:21

UPDATE: I have now reproduced the original issue, and it can be reproduced with the tristan/test-broken-track-all branch of the gnome-build-meta repository.

That branch is not expected to build successfully at all, but it will reproduce the issue so long as upstreams do not decide to vanish out of thin air...

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 11:17

mentioned in commit 23460139ce51c044bc5eb6d52858ca1f0c578a37

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 11:18

mentioned in merge request !1331

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 11:31

mentioned in commit d9d2ebfa97486a94b8080ef57f3b017f60efdc0e

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 11:44

mentioned in commit 848670e99fa0f301e0372ad74571d232734e9aeb

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 11:45

mentioned in merge request !1332

In GitLab by [Gitlab user @tristanvb] on May 8, 2019, 11:59

I have reduced this to a simplified test case and opened WIP merge requests:

bst-1.2 MR !1331 - currently adds isolated test case that fails
master MR !1332 - currently adds isolated test case that fails

In GitLab by [Gitlab user @tristanvb] on May 9, 2019, 07:32

mentioned in commit 210a4bfb9fa311ee256dc0cb7e44d52d73a2da86

In GitLab by [Gitlab user @tristanvb] on May 9, 2019, 09:20

mentioned in commit 223834422d12136879ea2840bd8c8270da87b657

In GitLab by [Gitlab user @tristanvb] on May 9, 2019, 09:20

mentioned in commit 70183d8274898701f4102a1581262807ca89f9ee

In GitLab by [Gitlab user @tristanvb] on May 9, 2019, 09:31

mentioned in commit 627502a63ab1e1cf1e878d33b9983df88f01e03f

In GitLab by [Gitlab user @tristanvb] on May 9, 2019, 09:31

mentioned in commit b17a5e071a5abc76f57aae1f7ab7194f55b40360

In GitLab by [Gitlab user @tristanvb] on May 10, 2019, 09:15

closed via merge request !1332

In GitLab by [Gitlab user @tristanvb] on May 10, 2019, 09:15

mentioned in commit 463ba24d1da14546a844b97fd6845aec4ce7b4cb

In GitLab by [Gitlab user @tristanvb] on May 11, 2019, 08:45

Looks like I was too quick on the trigger, and this issue was only partially fixed by !1332 and !1331

The test case seems to not reproduce the full issue, running the following command on the tristan/test-broken-track-all branch of gnome-build-meta:

bst --log-file --no-strict --on-error continue build --track-all vm/desktop-vm-image-x86_64.bst

Still gets me these results:

Pipeline Summary
  Total:       609
  Session:     609
  Track Queue: processed 229, skipped 13, failed 0 
  Pull Queue:  processed 0,   skipped 0,  failed 0 
  Fetch Queue: processed 0,   skipped 0,  failed 0 
  Build Queue: processed 0,   skipped 0,  failed 0

In GitLab by [Gitlab user @tristanvb] on May 11, 2019, 08:45

reopened

In GitLab by [Gitlab user @tristanvb] on May 11, 2019, 08:52

Oh this is strange.

I ran it again and the bug didn't reproduce.
I verified the previous log, and it is the correct version 1.2.6+17.g92555c4e which includes the fix.
I completely deleted the local artifact cache and ran it again, and the bug again did not reproduce.

When the issue does not reproduce, it does exactly what I expect: It tracks everything first before starting to pull vm/desktop-vm-image-x86_64.bst, as it needs to derive a strict cache key to pull with. Then after failing to pull, it decides that build-only dependencies need to be built, and goes ahead to pull more things.

I am sure the issue happened right in front of me, so let me keep this open for a bit and see if I can reproduce it again.

In GitLab by [Gitlab user @tristanvb] on May 11, 2019, 08:56

changed the description

In GitLab by [Gitlab user @tristanvb] on May 11, 2019, 09:26

I'll remove critical from here for now, as it seems this is hard to reproduce and is not necessarily preventing anyone from getting work done.

In GitLab by [Gitlab user @tlater] on Nov 29, 2019, 17:03

The --track option has been removed from bst build in master now, so this is an exclusively 1.4 issue now.

buildstream-migration / bst-staging