Invalid head? or figure out what happened here

[rhendric]

https://r.ryantm.com/log/stacks/2023-09-03.log

After waiting several hours for OfBorg to become free, the GitHub API rejected the pull request with a 422 and the following message:

{
  message: 'Validation Failed',
  errors: [ { resource: 'PullRequest', field: 'head', code: 'invalid' } ],
  documentation_url: 'https://docs.github.com/rest/pulls/pulls#create-a-pull-request'
}

We should probably be more robust against whatever this is.

[rhendric]

Saw a lot of these today:

https://r.ryantm.com/log/bruno/2023-10-08.log https://r.ryantm.com/log/castopod/2023-10-08.log https://r.ryantm.com/log/cpm-cmake/2023-10-08.log https://r.ryantm.com/log/datasette/2023-10-08.log https://r.ryantm.com/log/dt/2023-10-08.log https://r.ryantm.com/log/earthly/2023-10-08.log https://r.ryantm.com/log/eigenmath/2023-10-08.log https://r.ryantm.com/log/fioctl/2023-10-08.log

Waiting for OfBorg availability seems to be a common factor in all of these; maybe something is going stale somehow?

[rhendric]

I have a hypothesis: looking at branch activity, r-ryantm is creating these branches for each of these packages and then deleting them. These deletions are happening more than once daily, so initially I didn't suspect the nixpkgs-update-delete-done service, but given that this service is restarted every time the infrastructure is updated and today seems to have been a busy day for that, it does seem possible that each time that happened, any branches that were pushed but waiting for OfBorg availability got deleted, which would indeed make the head invalid.

I see two things we could do about this, not mutually exclusive:

• move the push to after the OfBorg wait, to reduce the window of time in which this could happen
• repush if the create PR call fails with a 422, to recover if it happens anyway