Branch indexing for linux-kernel equuleus is strange

[GurliGebis]

Just imported the jobs using the seed jobs bash file.

For sagitta, it works just fine, but for equuleus, when it does branch indexing, it starts pulling the non-local-built vyos-build container, and building the package.

Here is a shorted output of the branch indexing:

Branch indexing
 > git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-037eb4f9b5af4a4a58df3bf00a20621d/.git # timeout=10
Setting origin to https://github.com/vyos/vyos-build.git
 > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git config --get remote.origin.url # timeout=10
 > git fetch --tags --force --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/T3664-mixin-path
Seen branch in repository origin/circinus-stream
Seen branch in repository origin/crux
Seen branch in repository origin/current
Seen branch in repository origin/equuleus
Seen branch in repository origin/feature/T6386-add-caller-workflows-codeoowners
Seen branch in repository origin/feature/T6386-equuleus-add-caller-wf
Seen branch in repository origin/feature/T6386-sagitta-add-caller-wf
Seen branch in repository origin/feature/T6399-codeowner-correction-current
Seen branch in repository origin/feature/T6399-codeowner-correction-sagitta
Seen branch in repository origin/sagitta
Seen branch in repository origin/sagitta-stream
Seen 12 remote branches
Obtained packages/linux-kernel/Jenkinsfile from b3da0d46ea029b65a8853b3197374931703a4b60
Loading library vyos-build@equuleus
Attempting to resolve equuleus from remote references...
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git ls-remote -h -- https://github.com/dd010101/vyos-build.git # timeout=10
Found match: refs/heads/equuleus revision 3e2303c7f9225f21ffa06e900553691328addb2b
The recommended git tool is: git
No credentials specified
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/dd010101/vyos-build.git
 > git init /var/lib/jenkins/workspace/linux-kernel_equuleus@libs/480506e4968dfef5f81892d6cdacc2eccdf6a3d4d9a2b875ef98ee90a59a1bcd # timeout=10
Fetching upstream changes from https://github.com/dd010101/vyos-build.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git fetch --no-tags --force --progress -- https://github.com/dd010101/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/dd010101/vyos-build.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 3e2303c7f9225f21ffa06e900553691328addb2b (equuleus)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3e2303c7f9225f21ffa06e900553691328addb2b # timeout=10
Commit message: "added vyos-build-container job as pseudo-package"
 > git rev-list --no-walk 3e2303c7f9225f21ffa06e900553691328addb2b # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] properties
[Pipeline] node
Running on [Jenkins](http://10.10.1.97:8080/computer/(built-in)/) in /var/lib/jenkins/workspace/linux-kernel_equuleus
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Define Agent)
[Pipeline] script
[Pipeline] {
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] getContext
[Pipeline] node
Running on [Jenkins](http://10.10.1.97:8080/computer/(built-in)/) in /var/lib/jenkins/workspace/linux-kernel_equuleus
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: git
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/vyos/vyos-build.git
 > git init /var/lib/jenkins/workspace/linux-kernel_equuleus # timeout=10
Fetching upstream changes from https://github.com/vyos/vyos-build.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git fetch --tags --force --progress -- https://github.com/vyos/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision b3da0d46ea029b65a8853b3197374931703a4b60 (equuleus)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b3da0d46ea029b65a8853b3197374931703a4b60 # timeout=10
Commit message: "Merge pull request #633 from cjac/cjac-equuleus"
 > git rev-list --no-walk b3da0d46ea029b65a8853b3197374931703a4b60 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Agent Setup)
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker pull vyos/vyos-build:equuleus
Sending interrupt signal to process

At this point I interrupted it (I let it ran earlier, and it started to compile everything after doing the pull).

How can we fix it, so it just indexes it (like it should), instead of pulling the docker image (it shouldn't), and build the package (it shouldn't either).

Here is the output from branch indexing for the sagitta branch:

Branch indexing
 > git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-037eb4f9b5af4a4a58df3bf00a20621d/.git # timeout=10
Setting origin to https://github.com/vyos/vyos-build.git
 > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git config --get remote.origin.url # timeout=10
 > git fetch --tags --force --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/T3664-mixin-path
Seen branch in repository origin/circinus-stream
Seen branch in repository origin/crux
Seen branch in repository origin/current
Seen branch in repository origin/equuleus
Seen branch in repository origin/feature/T6386-add-caller-workflows-codeoowners
Seen branch in repository origin/feature/T6386-equuleus-add-caller-wf
Seen branch in repository origin/feature/T6386-sagitta-add-caller-wf
Seen branch in repository origin/feature/T6399-codeowner-correction-current
Seen branch in repository origin/feature/T6399-codeowner-correction-sagitta
Seen branch in repository origin/sagitta
Seen branch in repository origin/sagitta-stream
Seen 12 remote branches
Obtained packages/linux-kernel/Jenkinsfile from 258b865b685d8f92e75bcd96ab311d1d6abc96ff
Loading library vyos-build@sagitta
Attempting to resolve sagitta from remote references...
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git ls-remote -h -- https://github.com/dd010101/vyos-build.git # timeout=10
Found match: refs/heads/sagitta revision 14e6c50ad1b914bd278f859a14043d6974a94c80
The recommended git tool is: git
No credentials specified
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/dd010101/vyos-build.git
 > git init /var/lib/jenkins/workspace/linux-kernel_sagitta@libs/480506e4968dfef5f81892d6cdacc2eccdf6a3d4d9a2b875ef98ee90a59a1bcd # timeout=10
Fetching upstream changes from https://github.com/dd010101/vyos-build.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.2'
 > git fetch --no-tags --force --progress -- https://github.com/dd010101/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/dd010101/vyos-build.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 14e6c50ad1b914bd278f859a14043d6974a94c80 (sagitta)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 14e6c50ad1b914bd278f859a14043d6974a94c80 # timeout=10
Commit message: "added vyos-build-container job as pseudo-package"
First time build. Skipping changelog.
[Pipeline] Start of Pipeline
[Pipeline] timeout
Timeout set to expire in 4 hr 0 min
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Define Agent)
[Pipeline] node
15:25:48  Running on Jenkins in /var/lib/jenkins/workspace/linux-kernel_sagitta
[Pipeline] {
[Pipeline] checkout
15:25:50  The recommended git tool is: git
15:25:50  No credentials specified
15:25:50  Cloning the remote Git repository
15:25:50  Cloning repository https://github.com/vyos/vyos-build.git
15:25:50   > git init /var/lib/jenkins/workspace/linux-kernel_sagitta # timeout=10
15:25:50  Fetching upstream changes from https://github.com/vyos/vyos-build.git
15:25:50   > git --version # timeout=10
15:25:50   > git --version # 'git version 2.39.2'
15:25:50   > git fetch --tags --force --progress -- https://github.com/vyos/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
15:25:51   > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10
15:25:51   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
15:25:51  Avoid second fetch
15:25:51  Checking out Revision 258b865b685d8f92e75bcd96ab311d1d6abc96ff (sagitta)
15:25:51   > git config core.sparsecheckout # timeout=10
15:25:51   > git checkout -f 258b865b685d8f92e75bcd96ab311d1d6abc96ff # timeout=10
15:25:51  Commit message: "Merge pull request #650 from vyos/mergify/bp/sagitta/pr-649"
15:25:51  First time build. Skipping changelog.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] echo
15:25:59  Warning, empty changelog. Probably because this is the first build.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
Stage "Define Agent" skipped due to when conditional
[Pipeline] getContext
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build Code)
[Pipeline] echo
15:26:14  Warning, empty changelog. Probably because this is the first build.
Stage "Build Code" skipped due to when conditional
[Pipeline] getContext
[Pipeline] parallel
[Pipeline] { (Branch: amd64)
[Pipeline] { (Branch: arm64)
[Pipeline] stage
[Pipeline] { (amd64)
[Pipeline] stage
[Pipeline] { (arm64)
Stage "amd64" skipped due to when conditional
[Pipeline] getContext
[Pipeline] }
Stage "arm64" skipped due to when conditional
[Pipeline] getContext
[Pipeline] }
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Finalize)
[Pipeline] node
15:26:53  Running on Jenkins in /var/lib/jenkins/workspace/linux-kernel_sagitta
[Pipeline] {
[Pipeline] checkout
15:26:55  The recommended git tool is: git
15:26:55  No credentials specified
15:26:55   > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/linux-kernel_sagitta/.git # timeout=10
15:26:55  Fetching changes from the remote Git repository
15:26:55   > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10
15:26:55  Fetching upstream changes from https://github.com/vyos/vyos-build.git
15:26:55   > git --version # timeout=10
15:26:55   > git --version # 'git version 2.39.2'
15:26:55   > git fetch --tags --force --progress -- https://github.com/vyos/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
15:26:55  Checking out Revision 258b865b685d8f92e75bcd96ab311d1d6abc96ff (sagitta)
15:26:55   > git config core.sparsecheckout # timeout=10
15:26:55   > git checkout -f 258b865b685d8f92e75bcd96ab311d1d6abc96ff # timeout=10
15:26:55  Commit message: "Merge pull request #650 from vyos/mergify/bp/sagitta/pr-649"
[Pipeline] withEnv
[Pipeline] {
[Pipeline] echo
15:27:01  Warning, empty changelog. Probably because this is the first build.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
Stage "Finalize" skipped due to when conditional
[Pipeline] getContext
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline
Finished: SUCCESS

[GurliGebis]

Just checked the Jenkinsfile for the two branches: https://github.com/dd010101/vyos-build/blob/equuleus/packages/linux-kernel/Jenkinsfile https://github.com/dd010101/vyos-build/blob/sagitta/packages/linux-kernel/Jenkinsfile

It looks like the equuleus one is completely different from all other packages (only checked a few) - do we need to change it in your vyos-build repo, and change it to use that one instead?

[GurliGebis]

Commit when they changed it for current (back before sagitta was branched out from current): https://github.com/dd010101/vyos-build/commit/d127e81f0ca1ad7d358f376c0112a5c9c4d16a88

[dd010101]

linux-kernel equuleus does build after indexing by itself - I can confirm this isn't the case, I did delete the linux-kernel and add it back as new job and branch indexing didn't trigger build

linux-kernel equuleus uses wrong vyos-build docker container - I can confirm this isn't the case, I see that the build pulls what is should.

docker pull 172.17.17.17:5000/vyos/vyos-build:equuleus
equuleus: Pulling from vyos/vyos-build
Digest: sha256:418c6e91565a12a3ead7f59ac8fab7a9eaebdef97c07e178ef4561c2a3e2c925
Status: Image is up to date for 172.17.17.17:5000/vyos/vyos-build:equuleus
172.17.17.17:5000/vyos/vyos-build:equuleus

I also don't see the point of replacing the Jenkinsfile just because it's different - if there is something wrong then we should fix it not replace it - because of merging future changes.

Is there anything wrong with the Jenkinsfile though?

Why would the build start by itself? Well I don't know but I saw this before and I originally was thinking that Jenkins launches first build automatically but it doesn't - well most of the time anyway. I saw "auto build" with different random packages, it wasn't the kernel for equuleus, it was rare but it did happen.

Why would the build pull wrong vyos-build docker container? I can only guess but I would say that's makes sense because you don't have your local vyos-build built yet? Then docker will fallback to dockerhub.

That's why I would like to know if you can reproduce this again - if you create another linux-kernel pipeline (by hand) - will it launch build every time? Like three times in row? It doesn't matter if you add it by hand or the script that's why debugging you can use Jenkins GUI.

Here you see fresh first branch indexing of linux-kernel equuleus and it doesn't trigger build. It knows the condition wasn't met:

Branch indexing

``` Branch indexing > git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-037eb4f9b5af4a4a58df3bf00a20621d/.git # timeout=10 Setting origin to https://github.com/vyos/vyos-build.git > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 > git --version # 'git version 2.39.2' > git config --get remote.origin.url # timeout=10 > git fetch --tags --force --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Seen branch in repository origin/T3664-mixin-path Seen branch in repository origin/circinus-stream Seen branch in repository origin/crux Seen branch in repository origin/current Seen branch in repository origin/equuleus Seen branch in repository origin/feature/T6386-add-caller-workflows-codeoowners Seen branch in repository origin/feature/T6386-equuleus-add-caller-wf Seen branch in repository origin/feature/T6386-sagitta-add-caller-wf Seen branch in repository origin/feature/T6399-codeowner-correction-current Seen branch in repository origin/feature/T6399-codeowner-correction-sagitta Seen branch in repository origin/sagitta Seen branch in repository origin/sagitta-stream Seen 12 remote branches Obtained packages/linux-kernel/Jenkinsfile from b3da0d46ea029b65a8853b3197374931703a4b60 Loading library vyos-build@equuleus Attempting to resolve equuleus from remote references... > git --version # timeout=10 > git --version # 'git version 2.39.2' > git ls-remote -h -- https://github.com/dd010101/vyos-build.git # timeout=10 Found match: refs/heads/equuleus revision 3e2303c7f9225f21ffa06e900553691328addb2b WARNING: ignoring request to compute changelog in clone mode The recommended git tool is: git No credentials specified Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://github.com/dd010101/vyos-build.git > git init /var/lib/jenkins/jobs/linux-kernel2/branches/equuleus/builds/1/libs/cb6986ee395d3710a89e99c2dad0d388ee208e764c696736f9360a34103b8f5f/root # timeout=10 Fetching upstream changes from https://github.com/dd010101/vyos-build.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --no-tags --force --progress -- https://github.com/dd010101/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/dd010101/vyos-build.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 3e2303c7f9225f21ffa06e900553691328addb2b (equuleus) > git config core.sparsecheckout # timeout=10 > git checkout -f 3e2303c7f9225f21ffa06e900553691328addb2b # timeout=10 Commit message: "added vyos-build-container job as pseudo-package" [Pipeline] Start of Pipeline [Pipeline] properties [Pipeline] node Running on Jenkins in /var/lib/jenkins/workspace/linux-kernel2_equuleus [Pipeline] { [Pipeline] stage [Pipeline] { (Define Agent) [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] getContext [Pipeline] node Running on Jenkins in /var/lib/jenkins/workspace/linux-kernel2_equuleus [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Cloning repository https://github.com/vyos/vyos-build.git > git init /var/lib/jenkins/workspace/linux-kernel2_equuleus # timeout=10 Fetching upstream changes from https://github.com/vyos/vyos-build.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress -- https://github.com/vyos/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b3da0d46ea029b65a8853b3197374931703a4b60 (equuleus) > git config core.sparsecheckout # timeout=10 > git checkout -f b3da0d46ea029b65a8853b3197374931703a4b60 # timeout=10 Commit message: "Merge pull request #633 from cjac/cjac-equuleus" First time build. Skipping changelog. [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withDockerRegistry [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Agent Setup) [Pipeline] isUnix [Pipeline] withEnv [Pipeline] { [Pipeline] sh + docker pull 172.17.17.17:5000/vyos/vyos-build:equuleus equuleus: Pulling from vyos/vyos-build Digest: sha256:418c6e91565a12a3ead7f59ac8fab7a9eaebdef97c07e178ef4561c2a3e2c925 Status: Image is up to date for 172.17.17.17:5000/vyos/vyos-build:equuleus 172.17.17.17:5000/vyos/vyos-build:equuleus [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] isUnix [Pipeline] withEnv [Pipeline] { [Pipeline] sh + docker inspect -f . vyos/vyos-build:equuleus . [Pipeline] } [Pipeline] // withEnv [Pipeline] withDockerContainer Jenkins does not seem to be running inside a container $ docker run -t -d -u 1006:1006 --sysctl net.ipv6.conf.lo.disable_ipv6=0 -e GOSU_UID=1006 -e GOSU_GID=1006 -w /var/lib/jenkins/workspace/linux-kernel2_equuleus -v /var/lib/jenkins/workspace/linux-kernel2_equuleus:/var/lib/jenkins/workspace/linux-kernel2_equuleus:rw,z -v /var/lib/jenkins/workspace/linux-kernel2_equuleus@tmp:/var/lib/jenkins/workspace/linux-kernel2_equuleus@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** vyos/vyos-build:equuleus cat $ docker top c536ad83c95595121ee883f300a74425fc6e88a4212aabc0706b7a15bb3facb8 -eo pid,comm ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Alternatively you can force image entrypoint to be disabled by adding option `--entrypoint=''`. [Pipeline] { [Pipeline] sh + dpkg --print-architecture [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 2 hr 0 min [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] stage [Pipeline] { (Fetch) [Pipeline] script [Pipeline] { [Pipeline] checkout 19:05:11 The recommended git tool is: git 19:05:11 No credentials specified 19:05:11 Warning: JENKINS-30600: special launcher org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@1a0ec69b; decorates hudson.Launcher$LocalLauncher@1ae72ab will be ignored (a typical symptom is the Git executable not being run inside a designated container) 19:05:11 > git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/linux-kernel2_equuleus/.git # timeout=10 19:05:11 Fetching changes from the remote Git repository 19:05:11 > git config remote.origin.url https://github.com/vyos/vyos-build.git # timeout=10 19:05:11 Fetching upstream changes from https://github.com/vyos/vyos-build.git 19:05:11 > git --version # timeout=10 19:05:11 > git --version # 'git version 2.39.2' 19:05:11 > git fetch --tags --force --progress -- https://github.com/vyos/vyos-build.git +refs/heads/*:refs/remotes/origin/* # timeout=10 19:05:11 Checking out Revision b3da0d46ea029b65a8853b3197374931703a4b60 (equuleus) 19:05:11 > git config core.sparsecheckout # timeout=10 19:05:11 > git checkout -f b3da0d46ea029b65a8853b3197374931703a4b60 # timeout=10 19:05:11 Commit message: "Merge pull request #633 from cjac/cjac-equuleus" [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Git Clone - Components) [Pipeline] parallel [Pipeline] { (Branch: Kernel) [Pipeline] { (Branch: Kernel Firmware) [Pipeline] { (Branch: WireGuard) [Pipeline] { (Branch: Accel-PPP) [Pipeline] stage [Pipeline] { (Kernel) [Pipeline] stage [Pipeline] { (Kernel Firmware) [Pipeline] stage [Pipeline] { (WireGuard) [Pipeline] stage [Pipeline] { (Accel-PPP) [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:12 Warning, empty changelog. Probably because this is the first build. Stage "Kernel" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "Kernel Firmware" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "WireGuard" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "Accel-PPP" skipped due to when conditional [Pipeline] getContext [Pipeline] } [Pipeline] // stage [Pipeline] // stage [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Compile Kernel) [Pipeline] echo 19:05:13 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:13 Warning, empty changelog. Probably because this is the first build. Stage "Compile Kernel" skipped due to when conditional [Pipeline] getContext [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Kernel Module(s)) [Pipeline] parallel [Pipeline] { (Branch: WireGuard) [Pipeline] { (Branch: Accel-PPP) [Pipeline] { (Branch: Intel QuickAssist Technology) [Pipeline] { (Branch: Intel ICE driver) [Pipeline] { (Branch: Realtek r8152 driver) [Pipeline] stage [Pipeline] { (WireGuard) [Pipeline] stage [Pipeline] { (Accel-PPP) [Pipeline] stage [Pipeline] { (Intel QuickAssist Technology) [Pipeline] stage [Pipeline] { (Intel ICE driver) [Pipeline] stage [Pipeline] { (Realtek r8152 driver) [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:14 Warning, empty changelog. Probably because this is the first build. Stage "WireGuard" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "Accel-PPP" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "Intel QuickAssist Technology" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "Intel ICE driver" skipped due to when conditional [Pipeline] getContext [Pipeline] } Stage "Realtek r8152 driver" skipped due to when conditional [Pipeline] getContext [Pipeline] } [Pipeline] // stage [Pipeline] // stage [Pipeline] // stage [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Linux Firmware) [Pipeline] echo 19:05:15 Warning, empty changelog. Probably because this is the first build. [Pipeline] echo 19:05:15 Warning, empty changelog. Probably because this is the first build. Stage "Linux Firmware" skipped due to when conditional [Pipeline] getContext [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] script [Pipeline] { [Pipeline] dir 19:05:15 Running in /var/lib/jenkins/workspace/linux-kernel2_equuleus/packages/linux-kernel [Pipeline] { [Pipeline] sshagent 19:05:15 [ssh-agent] Using credentials jenkins 19:05:15 [ssh-agent] Looking for ssh-agent implementation... 19:05:15 [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) 19:05:15 $ docker exec c536ad83c95595121ee883f300a74425fc6e88a4212aabc0706b7a15bb3facb8 ssh-agent 19:05:15 SSH_AUTH_SOCK=/tmp/ssh-KTakCkXynr3y/agent.43 19:05:15 SSH_AGENT_PID=49 19:05:15 Running ssh-add (command line suppressed) 19:05:15 Identity added: /var/lib/jenkins/workspace/linux-kernel2_equuleus/packages/linux-kernel@tmp/private_key_12455626030372779197.key (jenkins) 19:05:15 [ssh-agent] Started. [Pipeline] { [Pipeline] echo 19:05:15 Uploading package(s) and updating package(s) in the repository ... [Pipeline] findFiles [Pipeline] } 19:05:15 $ docker exec --env ******** --env ******** c536ad83c95595121ee883f300a74425fc6e88a4212aabc0706b7a15bb3facb8 ssh-agent -k 19:05:15 unset SSH_AUTH_SOCK; 19:05:15 unset SSH_AGENT_PID; 19:05:15 echo Agent pid 49 killed; 19:05:15 [ssh-agent] Stopped. [Pipeline] // sshagent [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // script [Pipeline] deleteDir [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } $ docker stop --time=1 c536ad83c95595121ee883f300a74425fc6e88a4212aabc0706b7a15bb3facb8 $ docker rm -f --volumes c536ad83c95595121ee883f300a74425fc6e88a4212aabc0706b7a15bb3facb8 [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withDockerRegistry [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS ```

[dd010101]

Jenkins should launch auto-build only if it detects that: 1) new commit happened (it compares previous last commit and current last commit hash) 2) the commit contains changed file by specific pattern and this pattern is specified in the Jenskinfile in question by changeset pattern - https://github.com/dd010101/vyos-build/blob/equuleus/packages/linux-kernel/Jenkinsfile#L83-L84

Both of those conditions need to be true. Thus that's why I can't explain why any package would do that when added and I don't think you can repeat it. The linux-kernel has these conditions as well - same conditions as other packages. The only other option is triggeredBy cause: "UserIdCause" - if the user launches build.

Maybe there is another possibility - if the last commit matched the changeset pattern - then you would trigger the build even if no commit changed since Jenkins runs the first time thus has nothing to compare with? But last commit doesn't match the patterns so this doesn't explain what you see, but it could explain what I saw in past. I'm not sure if Jenkins has condition for this case where it would skip because the previous hash is null.

What we really want is to not do any actual build under any condition or we want to queue actual build after the indexing. The latter I don't know how to do and the former I know but that would require to modify every standalone Jenskinfile and shared library and that's a lot of changes.

[GurliGebis]

I do have the containers built, it was run using the vyos-build-container job first, after that I imported the jobs 🙂 Waiting for the initial provision of jobs is easy enough, my install scripts does that already. The problem is that the initial import causes the build for some reason. I will try wiping the jenkins install and retry, just in case something is wrong, and get back.

[GurliGebis]

Running docker image ls confirms the image already exists on the machine (the one I built) So I don't know why Jenkins does this, but it is strange

[dd010101]

The pull means the Jenkins didn't have Declarative Pipeline (Docker) I guess. Otherwise you would see:

docker pull 172.17.17.17:5000/vyos/vyos-build:equuleus

instead of yours:

docker pull vyos/vyos-build:equuleus

The pull should be skipped normally because the condition isn't met:

Stage "Define Agent" skipped due to when conditional

So the pull seems like wrong settings but it shouldn't run and it doesn't run for me.

You don't need to wipe everything - just add linux-kernel2 and copy linux-kernel and this is the same thing as you would do with the script - the branch indexing gets triggered on empty pipeline.

What I saw was for sure fluke - I wasn't able to repeat it ever again with the same job - it happened randomly.

You could modify all conditions to include for example environment variable that would neutralize the build the same way how the ARM64_BUILD_DISABLED works - you would set DISABLE_ALL_BUILDS=true before you add jobs and if you know the indexing is done then set DISABLE_ALL_BUILDS=false and this way you know there is no way to trigger the build by fluke - even if it gets triggered it will be skipped due to the variable. But this isn't one condition in one place, at least 5 or more places have this condition so if there is other global way without modifying the source code that would be better.

[GurliGebis]

I'll take a look at the settings - these are the images on the machine:

docker image ls
REPOSITORY                          TAG        IMAGE ID       CREATED       SIZE
172.17.17.17:5000/vyos/vyos-build   equuleus   7adf8e5adeca   5 hours ago   4.34GB
vyos/vyos-build                     equuleus   7adf8e5adeca   5 hours ago   4.34GB
vyos/vyos-build                     current    5e1a58e14e81   6 hours ago   2.95GB
172.17.17.17:5000/vyos/vyos-build   current    5e1a58e14e81   6 hours ago   2.95GB
vyos/vyos-build                     sagitta    d3b28d56cd86   6 hours ago   2.48GB
172.17.17.17:5000/vyos/vyos-build   sagitta    d3b28d56cd86   6 hours ago   2.48GB
registry                            2.7        b8604a3fe854   2 years ago   26.2MB

[GurliGebis]

Indeed you are right - I haven't gotten Declarative Pipeline (Docker) set up 🙂 One more step to add to my install script - luckily, once I'm done, no one will make my mistakes 😀

[dd010101]

Is the random automatic build even a issue then? If you correct the docker pull then the early build isn't really a problem right?

[GurliGebis]

It depends, I'll test it soon (right now I'm using hostapd to verify that it can build packages). The idea is that it provisions all the jobs, waits for them to be done, and once they are done, it starts X number of jobs at a time (where X is the number of CPU cores and executors on the server). So if it starts to build the kernel, that will slow down that part quiet a bit - but I'll see if it works now and get back to you.

If it does, I'm 95% done with the installer, up to the point of it resulting in all packages being built at the end. Then I just need a few steps, so you can ask it to build an ISO for you.

[dd010101]

👍

Currently you wait for all indexing to complete before triggering build step? Thus unexpected complication would make the indexing step long?

There is also option to think about different scheduling principle - to be state aware. One could track what package completed indexing and one could start build after the indexing for the package is completed - per package - this way there is no waiting. One could limit that X concurrent indexing/builds are active at one time. Ideally one would want to track if both indexing was done and if the package was built by the indexing and then one wouldn't even care if the build would trigger by itself. What about adding report what build failed as well? Maybe even retry failed build for second time to give a chance to fix itself? Should I be the one who writes such scheduler? 🤔

[GurliGebis]

I think it would make sense for me to do what I'm working on, and then once I'm done, you can take it and run with it, to rewrite it if you want 🙂

[GurliGebis]

But yes, I'm waiting for them to finish provisioning before I trigger the actual initial build.

[GurliGebis]

But all that is a job for another day - I'm off for today 🙂

[GurliGebis]

It seems to have resolved itself.

Here is a small treat - I can provision all the packages and wait for them to finish with branch indexing 😀

Please enter you username here: GurliGebis
Please enter your Jenkins token here: REDACTED

Testing jenkins connection: OK

Provisioning jobs in Jenkins...
aws-gateway-load-balancer-tunnel-handler: OK
ddclient: OK
dropbear: OK
ethtool: OK
frr: OK
hostap: OK
hsflowd: OK
hvinfo: OK
ipaddrcheck: OK
iproute2: OK
isc-dhcp: OK
keepalived: OK
libnss-mapuser: OK
libnss-tacplus: OK
libpam-radius-auth: OK
libvyosconfig: OK
linux-kernel: OK
live-boot: OK
mdns-repeater: OK
minisign: OK
ndppd: OK
netfilter: OK
ocserv: OK
opennhrp: OK
openvpn-otp: OK
owamp: OK
pam_tacplus: OK
pmacct: OK
pyhumps: OK
python3-inotify: OK
radvd: OK
strongswan: OK
telegraf: OK
udp-broadcast-relay: OK
vyatta-bash: OK
vyatta-biosdevname: OK
vyatta-cfg-firewall: OK
vyatta-cfg-qos: OK
vyatta-cfg-quagga: OK
vyatta-cfg-system: OK
vyatta-cfg-vpn: OK
vyatta-cfg: OK
vyatta-cluster: OK
vyatta-config-mgmt: OK
vyatta-conntrack: OK
vyatta-nat: OK
vyatta-op-firewall: OK
vyatta-op-qos: OK
vyatta-op-vpn: OK
vyatta-op: OK
vyatta-wanloadbalance: OK
vyatta-zone: OK
vyos-1x: OK
vyos-cloud-init: OK
vyos-http-api-tools: OK
vyos-nhrp: OK
vyos-opennhrp: OK
vyos-strongswan: OK
vyos-user-utils: OK
vyos-utils: OK
vyos-world: OK
vyos-xe-guest-utilities: OK
wide-dhcpv6: OK

Waiting for jobs to be provisioned in Jenkins...
Jobs has been provisioned.

root@vyos-tester:~/vyos-jenkins#

[GurliGebis]

I have btw. created a PR that updates the jobs.json file to only contain the actual branches that is being built: #22

[dd010101]

Good job! Looks you are close to have everything? What you still missing/want to do?

[GurliGebis]

My todo list consists of:

• Have it build X number of packages at a time (got all the bits and pieces, just need to put it together)
• Create a script to trigger a complete ISO build
• General cleanup
• Documentation

[GurliGebis]

And also a final script to setup nginx (that's the easy part), so the iso builder can find the deb packages

[dd010101]

It looks like the equuleus one is completely different from all other packages (only checked a few) - do we need to change it in your vyos-build repo, and change it to use that one instead?

As luck has it - the Jenkinsfile in sagitta is the wrong one. Look at the Jenkinsfile of equuleus, you can see two changeset patterns, one **/packages/linux-kernel/* and the other one **/data/defaults.json. When they ported this non-standard Jenkinsfile to the standard buildPackage, then they met a challenge - the buildPackage doesn't support multiple changeset patterns, so how did they solve that? They didn't, they used only the first pattern and that's wrong. Today sagitta ISO build failed for me, since it didn't find kernel it expected to find and sure enough, they did update the kernel to 6.6.33 but that matches only the second pattern they dropped, so the Jenkins didn't know it and so I did have the previous 6.6.32 instead... Isn't that funny coincidence? 😄

[GurliGebis]

I'm at a loss for words

[dd010101]

I did found workaround for the changeset linux-kernel sagitta issue, so now linux-kernel needs to use fork (https://github.com/dd010101/vyos-build.git) as well. I don't have easy way to test this - we shall see!

[GurliGebis]

Nice 😊

Check this out - building in batches....

Please enter you username here: GurliGebis
Please enter your Jenkins token here: REDACTED

[  Completed  ] Package: libnss-tacplus - Branch: sagitta
[  Completed  ] Package: vyatta-op-vpn - Branch: equuleus
[  Completed  ] Package: vyatta-bash - Branch: equuleus
[  Completed  ] Package: vyatta-bash - Branch: sagitta
[  Completed  ] Package: radvd - Branch: sagitta
[  Completed  ] Package: vyatta-config-mgmt - Branch: equuleus
[  Completed  ] Package: vyos-cloud-init - Branch: equuleus
[  Completed  ] Package: vyos-cloud-init - Branch: sagitta
[  Completed  ] Package: telegraf - Branch: equuleus
[  Completed  ] Package: telegraf - Branch: sagitta
[  Completed  ] Package: pam_tacplus - Branch: sagitta
[  Completed  ] Package: minisign - Branch: equuleus
[  Completed  ] Package: hsflowd - Branch: sagitta
[  Completed  ] Package: vyatta-cfg-quagga - Branch: equuleus
[  Completed  ] Package: ddclient - Branch: sagitta
[  Completed  ] Package: vyatta-cfg-firewall - Branch: equuleus
[  Completed  ] Package: opennhrp - Branch: sagitta
[  Completed  ] Package: vyatta-biosdevname - Branch: equuleus
[  Completed  ] Package: vyatta-biosdevname - Branch: sagitta
[  Completed  ] Package: vyatta-cfg - Branch: equuleus
[  Completed  ] Package: vyatta-cfg - Branch: sagitta
[  Completed  ] Package: iproute2 - Branch: equuleus
[  Completed  ] Package: vyatta-conntrack - Branch: equuleus
[  Completed  ] Package: mdns-repeater - Branch: equuleus
[  Completed  ] Package: keepalived - Branch: equuleus
[   Running   ] Package: keepalived - Branch: sagitta
[   Running   ] Package: netfilter - Branch: equuleus
[   Running   ] Package: netfilter - Branch: sagitta
[  Completed  ] Package: vyatta-cfg-system - Branch: equuleus
[  Completed  ] Package: vyatta-cfg-system - Branch: sagitta

[GurliGebis]

(It is a snapshot of how far along it is, it takes MAX(4, CPU_CORES) tasks at a time

[GurliGebis]

All packages build fine using this method 😊

These are the kernel images I get out of it today (with it being built from your repo):

./public_html/repositories/sagitta/pool/main/v/vyos-linux-firmware
./public_html/repositories/sagitta/pool/main/v/vyos-linux-firmware/vyos-linux-firmware_20231211_all.deb
./public_html/repositories/sagitta/pool/main/l/linux-upstream
./public_html/repositories/sagitta/pool/main/l/linux-upstream/linux-headers-6.6.33-amd64-vyos_6.6.33-1_amd64.deb
./public_html/repositories/sagitta/pool/main/l/linux-upstream/linux-libc-dev_6.6.33-1_amd64.deb
./public_html/repositories/sagitta/pool/main/l/linux-upstream/linux-image-6.6.33-amd64-vyos_6.6.33-1_amd64.deb
./public_html/repositories/equuleus/pool/main/v/vyos-linux-firmware
./public_html/repositories/equuleus/pool/main/v/vyos-linux-firmware/vyos-linux-firmware_20201218_all.deb
./public_html/repositories/equuleus/pool/main/w/wireguard-linux-compat
./public_html/repositories/equuleus/pool/main/w/wireguard-linux-compat/wireguard-modules_1.0.20201112-1~bpo10+1_all.deb
./public_html/repositories/equuleus/pool/main/l/linux-5.4.268-amd64-vyos
./public_html/repositories/equuleus/pool/main/l/linux-5.4.268-amd64-vyos/linux-libc-dev_5.4.268-1_amd64.deb
./public_html/repositories/equuleus/pool/main/l/linux-5.4.268-amd64-vyos/linux-image-5.4.268-amd64-vyos_5.4.268-1_amd64.deb
./public_html/repositories/equuleus/pool/main/l/linux-5.4.268-amd64-vyos/linux-headers-5.4.268-amd64-vyos_5.4.268-1_amd64.deb
./public_html/repositories/equuleus/pool/main/l/linux-5.4.268-amd64-vyos/linux-tools-5.4.268-amd64-vyos_5.4.268-1_amd64.deb

[dd010101]

Check this out - building in batches....

Very nice!

Could you make CONCURRENT_JOBS_COUNT overridable by the user and include note in the echo "Concurrent jobs: $CONCURRENT_JOBS_COUNT" that user can change this? If for example the user encounters his system is too slow to handle the calculated number?

[GurliGebis]

Sure, I'll write a note about it - I'll let user set it using an environment variable

[GurliGebis]

Done 😊

[GurliGebis]

Thats it for now, might be a few days before I get time to work on the next steps. But next step is NGINX - we're getting close to the finish line now.

[GurliGebis]

@dd010101 look what I got 😊

####################################
# Unofficial VyOS ISO builder v1.0 #
####################################

Please enter which branch you want to build (equuleus or sagitta): sagitta
Please enter your email address: REDACTED

Cloning the VyOS build repository...
Checking out the sagitta branch...
Downloading apt signing key...
Building the ISO...

ISO build is complete.
The file is called: vyos-1.4-release-20240617-iso-amd64.iso.

Cleaning up...

[dd010101]

Nice, we can continue in the other thread if everything is done here!