search

NASA-PDS / roundup-action

Do a "roundup", a/k/a PDS-style continuous integration and delivery
Apache License 2.0
1 stars 4 forks source link

git push in roundup action failing w/ remote ahead of local #98

Closed jimmie closed 1 year ago

jimmie commented 1 year ago

🐛 Describe the bug

Trying to build harvest release/3.7.1 and roundup-action failed with following error:

1 file changed, 1 insertion(+), 1 deletion(-) DEBUG:pds.roundup.util:Stderr = «» DEBUG:pds.roundup.util:🏃‍♀️ Running «['git', 'push', 'origin', 'HEAD:main']» CRITICAL:pds.roundup.util:💥 Process with command line ['git', 'push', 'origin', 'HEAD:main'] failed with status 1 CRITICAL:pds.roundup.util:🪵 Stdout = «» CRITICAL:pds.roundup.util:📚 Stderr = «To https://github.com/NASA-PDS/harvest ! [rejected] HEAD -> main (fetch first) error: failed to push some refs to 'https://github.com/NASA-PDS/harvest' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. » Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/pds.roundup-0.0.0-py3.9.egg/pds/roundup/util.py", line 52, in invoke File "/usr/local/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['git', 'push', 'origin', 'HEAD:main']' returned non-zero exit status 1.

Could be a side effect of #76.

jordanpadams commented 1 year ago

setting severity to low for the time being unless it becomes a major problem moving forward. I think this usually just means you are tagging without the latest commit. I have noticed this happening when I merge a PR and then try to tag a release before the unstable build completes. The unstable build then sneaks in a change log update before the stable build completes.

jordanpadams commented 1 year ago

@nutjob4life @jimmie FYI ☝️

nutjob4life commented 1 year ago

Apparently there are two causes of the issue:

  1. In mid-Roundup, someone merges to main which causes Roundup's attempt to push to main to fail because its local tree is out of date, which makes sense.
  2. For some reason we can't fathom, a push to main fails because the local tree is out of date, but git pull says Already up to date anyway.

Putting a git pull in front of any push to HEAD:main sounds like an easy fix—except since we changed to triggering a release by a tag instead of a branch, we're always in a "detached HEAD" state in the Roundup action in stable releases.

Now, we could have the Roundup create a temporary branch from the release/X.Y.Z tag in which to do its work—and then a git pull would succeed. But that's a bigger underlying change to the Roundup and we'd want to vet this "fix" fully in the sandbox.

But given this is a rare occurrence and the issue itself is marked s.low maybe we should skip it? Thoughts?

jimmie commented 1 year ago

I am OK with kicking this to the curb.

nutjob4life commented 1 year ago

Over Slack, @jimmie wrote:

it seems we can reproduce that build error fairly easily - each time we’ve done a commit just before tagging - the subsequent build appears to fail consistently. Food for thought.

So not kicking this to the curb.

nutjob4life commented 1 year ago

The timeline that causes the problem seems to be as follows:

  1. git add some-file.c && git push origin main or merge a PR into main
  2. This triggers unstable-cicd.yaml which starts up a Roundup Action using the "unstable" assembly. What's happening at this point? _Quite a bit, including pushes directly into main.
  3. git tag release/1.2.3 && git push origin release/1.2.3
  4. This triggers stable-cicd.yaml which starts up a Roundup Action using the "stable" assembly. But the unstable assembly is still going on.

I think we need a mutex between Roundups so that one is ever only running at a time in a repository.

Thankfully, GitHub Actions provides such a feature for us: just add concurrency: roundup to all stable-cicd.yaml and unstable-cicd.yaml files at the workflow level. Sound good?

nutjob4life commented 1 year ago

The following repositories have pull requests pending may address the mutex (but may also have overlap with this issue):

jordanpadams commented 1 year ago

all PRs merged. done!