Git build step trying to get a commit from the wrong repository (possibly a bug)

[FatmanUK]

Python version: 3.7.8 Buildbot version: 2.7.0 Twisted version: 20.3.0 OS: FreeBSD 12.1-RELEASE-p8

Both workers --- tinyfeng and doclib --- were working fine, but after a commit to doclib's repo, the tinyfeng worker is suddenly trying to get that commit, even though it's not pointing at the same repo. The repos are hosted on the same Gitea server at \. The doclib repo is private hence the \:\ before the URL.

I could work around this issue with a shell command ("git pull") but I'd rather use the Git module if I can. Am I doing something amazingly stupid or is this a bug in the Git build step module? How is it confusing commits?

Here are my build steps from /var/buildbot/mother/config:

fc_doclib = util.BuildFactory()
fc_doclib.addStep(steps.Git(repourl='https://<user>:<password>@<url>:3000/Dreamtrack/doclib', mode='incremental'))
fc_doclib.addStep(steps.ShellCommand(name="build", command=["./build.sh"]))
bd_doclib = util.BuilderConfig(name="doclib", workernames=["doclib"], factory=fc_doclib)
c['builders'].append(bd_doclib)

fc_tinyfeng = util.BuildFactory()
fc_tinyfeng.addStep(steps.Git(repourl='https://<url>:3000/Dreamtrack/tinyfeng', mode='incremental'))
fc_tinyfeng.addStep(steps.ShellCommand(name="clean", command=["./clean.sh"], decodeRC={0:SUCCESS,1:SUCCESS}))
fc_tinyfeng.addStep(steps.ShellCommand(name="generate", command=["tup", "generate", "build.sh"]))
fc_tinyfeng.addStep(steps.ShellCommand(name="build", command=["./build.sh"]))
bd_tinyfeng = util.BuilderConfig(name="tinyfeng", workernames=["tinyfeng"], factory=fc_tinyfeng)
c['builders'].append(bd_tinyfeng)

Here is the output from tinyfeng worker:

git --version
 in dir /var/buildbot/tinyfeng/tinyfeng/build (timeout 1200 secs)
 watching logfiles {}
 argv: [b'git', b'--version']
 environment:
  LC_CTYPE=C.UTF-8
  PATH=/command:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  PWD=/var/buildbot/tinyfeng/tinyfeng/build
  SHLVL=0
 using PTY: False
git version 2.27.0
program finished with exit code 0
elapsedTime=0.039907
stat: No such file or directory: /var/buildbot/tinyfeng/tinyfeng/build/.buildbot-patchedprogram finished with exit code 2
program finished with exit code 0
git cat-file -e e436f4fe1ce92412ef982482f55cca2b30cd562c
 in dir /var/buildbot/tinyfeng/tinyfeng/build (timeout 1200 secs)
 watching logfiles {}
 argv: [b'git', b'cat-file', b'-e', b'e436f4fe1ce92412ef982482f55cca2b30cd562c']
 environment:
  LC_CTYPE=C.UTF-8
  PATH=/command:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  PWD=/var/buildbot/tinyfeng/tinyfeng/build
  SHLVL=0
 using PTY: False
program finished with exit code 1
elapsedTime=0.039196
git fetch -t https://<url>:3000/Dreamtrack/tinyfeng master --progress
 in dir /var/buildbot/tinyfeng/tinyfeng/build (timeout 1200 secs)
 watching logfiles {}
 argv: [b'git', b'fetch', b'-t', b'https://<url>:3000/Dreamtrack/tinyfeng', b'master', b'--progress']
 environment:
  LC_CTYPE=C.UTF-8
  PATH=/command:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  PWD=/var/buildbot/tinyfeng/tinyfeng/build
  SHLVL=0
  using PTY: False
From https://<url>:3000/Dreamtrack/tinyfeng
 * branch            master     -> FETCH_HEAD
program finished with exit code 0
elapsedTime=0.332584
git reset --hard e436f4fe1ce92412ef982482f55cca2b30cd562c --
 in dir /var/buildbot/tinyfeng/tinyfeng/build (timeout 1200 secs)
 watching logfiles {}
 argv: [b'git', b'reset', b'--hard', b'e436f4fe1ce92412ef982482f55cca2b30cd562c', b'--']
 environment:
  LC_CTYPE=C.UTF-8
  PATH=/command:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
  PWD=/var/buildbot/tinyfeng/tinyfeng/build
  SHLVL=0
 using PTY: False
fatal: Could not parse object 'e436f4fe1ce92412ef982482f55cca2b30cd562c'.
program finished with exit code 128
elapsedTime=0.052137

[p12tic]

Thanks for the bug report. It indeed looks like a regression in how we process changes. I'll look into it.

[terencehonles]

@FatmanUK how are your schedulers defined? (you shared the builders, but my hunch is it might be on that side).

I don't think I have any schedulers that span repositories, but my hunch would be something about the scheduler config would be leaking a revision to both builders and now that a revision is set both builders would be looking for that SHA which as you pointed out doesn't exist in that repo.

I could be totally wrong, but until @p12tic has further ideas it might make sense to look there.