`berkeley`: GHA workflow on `berkeley` branch checks out wrong Git commit

[eecavanna]

During the 2024.9 release (on Monday, September 23, 2024), @pkalita-lbl and I were trying to figure out why the GHA workflow (python-app.yml) that runs the unit tests was failing "on the tip of the berkeley branch" even though no relevant changes had been made on the berkeley branch since the workflow had last run successfully—the only changes that had been made were to a migration notebook, which does not affect the unit tests.

After digging in, @pkalita-lbl noticed that the GHA workflow was checking out a different commit than the one we had expected it to (we had expected it to check out the latest commit on the berkeley branch).

Here's a sequence of screenshots demonstrating this:

Screenshot
1. Look at the Git commit hash of the latest commit on the berkeley branch. It's 9cb5850.
2. Click the red "X" for the commit and click "Details" next to the failing check
3. Look at the Git commit hash in the "checkout" step. It's aa26394, as opposed to 9cb5850.

This ticket represents the task of getting to the bottom of why the GHA workflow on the berkeley branch is checking out a commit that is not the one we expect it to.

[eecavanna]

GitHub shows commit aa26394 here: https://github.com/microbiomedata/nmdc-runtime/commits/aa26394

Commit aa26394 immediately follows 9cb5850 on this "chain" of commits, which I don't think is a named branch.

Commit aa26394 contains a lot of changes with respect to 9cb5850!

[eecavanna]

According to GitHub (see second screenshot in the message above), the two parents of commit aa26394 are:

• 588287e (this is the latest commit on the main branch)
• 9cb5850 (this is the latest commit on the berkeley branch)

So, the GHA workflow was checking out a commit — not on any branch — where the berkeley branch had been merged into the main branch. Interesting! That answers the question of what it checked out.

Next steps:

• Find out why it checked that out
• Prevent it from doing that in the future

[eecavanna]

• Find out why it checked that out

I think I have a clue. I made an innocuous commit to the berkeley branch just now and then confirmed the GHA workflow was once again checking out a different commit. I noticed that the title on the GHA page was from a "DO NOT MERGE: ..." PR that I had created months ago to prepare for the eventual merge of the berkeley branch into the main branch. When I visited that PR page, I saw that it showed "Checks" running on the commit I made (or so I thought). When I clicked the orange bubble and looked at the commit being processed by GHA, however, I saw that different commit hash there.

So, I suspect the fact that I have a PR open where a merge from berkeley to main is being staged, is playing a role here. I may try closing that PR (it can always be re-opened or recreated) and seeing whether that has an effect on this symptom.