"Rerun Prebuild" button does not actually re-run failed prebuild

[jankeromnes]

Bug description

A Gitpod customer reports the following:

• Prebuilds are enabled for all branches on a project
• A new commit 1234 was pushed to branch-a
• A new prebuild abcd was triggered for commit 1234 on branch-a, but eventually the prebuild failed
• However, clicking on the button Rerun Prebuild (branch-a) does not actually re-run the prebuild -- instead, it redirects to the list of prebuilds

Investigating a bit further, we see that:

• The prebuild abcd appears in Failed state, with the error The tasks executed in the prebuild returned a non-zero exit code. headless task failed: exit status 255
• The prebuild abcd also does not show any logs (maybe another, unrelated bug?)
• When receiving the request triggerPrebuild (with commit ID 1234 and branch branch-a), Gitpod's server responds immediately with:

{
    result: {
        done: true,
        prebuildId: "abcd",
        wsid: "org-project-123"
    }
}

I guess Gitpod considers that commit 1234 already has a successful prebuild, which does not need to be rebuilt, even though prebuild abcd is actually Failed?

Steps to reproduce

1. Have prebuilds enabled for all branches in a project
2. Have a "Failed" prebuild for the latest commit of a given branch
3. Click on "Rerun Prebuild (branch-a)" in the bottom bar of the failed prebuild view

Workspace affected

No response

Expected behavior

Expected: Gitpod triggers a new prebuild for the branch, and then shows you its live logs and status

Actual: Gitpod redirects to the list of prebuilds, just as if the latest prebuild had been successful

Example repository

No response

Anything else?

No response

Front conversations

[jankeromnes]

In the database, we can see that prebuild abcd has:

• state: "available"
• error: "headless task failed: exit status 255"

Thus, I suspect that Gitpod returns this prebuild as "already completed successfully" just by looking at the state, whereas it should probably look at the error field as well.

[jankeromnes]

Another data point -- when opening the Prebuild page, it seems to show inconsistent status and logs between reloads:

• Sometimes it shows all the logs, and a successful (green) status
• Sometimes it doesn't show any logs, and a failed (red) status

[geropl]

@jankeromnes Let's separate issues out here:

• The prebuild at hand was run successfully (from the technical perspective), but the user tasks exited with "255". This is why it's marked as "available", which is IMO "correct": User's should be served this prebuild when starting a workspace based on it, so they can inspect why it failed in the first place
• The question is why the user expected to fix something with a re-trigger. And maybe it make sense to have a "Force rebuild" here (to handle out-of-control things, like build failed due to temporary registry failure), and ignore the existing prebuild - I think this is what this issue is about 💯

Sometimes it doesn't show any logs, and a failed (red) status

Let's handle that separately, and do a bit more research before we create a "this does not work" issue. There is known flakiness in how we can retrieve prebuild logs because our domain model relies on checks against to GitHub/GitLab to decide whether you can access a certain resource - and failing to do so would result in this behavior. ☝

[rgoldfinger-quizlet]

Hi, Jan was kind enough to work with me in diagnosing the three bugs here. I'll restate them for clarity:

1. Clicking the "re-run prebuild" button does not re-run the prebuild. I understand that you're saying that it should not re-run since it was "successful" by some measure, but if that is the case, then the UI should indicate as such, perhaps by re-naming the button. Currently it appears broken because the stated action fails silently, where clicking on the "re-run prebuild" button returns you to the list of prebuilds.

2. No logs for prebuild. This is one reason why I wanted to re-run the prebuild -- to see what happened. Separately, having logs for all prebuild tasks would be extremely helpful to say the least. https://github.com/gitpod-io/gitpod/issues/9994

3. When the prebuild failed, at first it said it was successful, and only showed that it errored after reloading the page. Initial:

   

After reloading:

User's should be served this prebuild when starting a workspace based on it, so they can inspect why it failed in the first place

This does not happen. Because the prebuild failed, workspaces do not use the prebuild. I confirmed this by checking the database row for the workspace and see that basedOnPrebuildId is empty. Perhaps this is a fourth bug.

[KayakinKoder]

Being able to force a rebuild, without conditions, would have saved us a lot of time on our team's initial gitpod setup (and would save us time going forward).

"Without conditions" meaning even if the prebuild succeeded, and even if the commit is the same. Why? External dependencies. For example, we update our custom docker image with some important new thing; currently, we have to push a fake commit to every single existing branch of code to trigger a new prebuild.

[gtsiolis]

Thanks for the feedback, @KayakinKoder! FWIW, we did some design exploration in the past for adding something like this to allow users to re-trigger a prebuild for any branch.

Re-posting here for visibility but I'll also circulate your feedback internally:

Early design on allowing users to re-trigger a prebuild

[jmls]

we're hitting this a number of times. What is the status ?

[jankeromnes]

Hi @jmls! This issue has not been scheduled to be worked on yet.

Meanwhile, are you able to use the workaround & just push any new commit to the same branch in order to reliably trigger a new prebuild?

[jmls]

oh that's an embuggerance.

Not exactly a great workaround if a developer has to modify source code / add tags just to force a prebuild again :(

[jldec]

I would agree that forcing users to push commits in order to re-run failed prebuilds is not great.

[KayakinKoder]

With gitpod doubling prices, we've had to entirely rework prebuilds. The fact that this issue is still open is making it nearly impossible for us to use gitpod in any appreciable manner.