search

gitpod-io / gitpod

The developer platform for on-demand cloud development environments to create software faster and more securely.
https://www.gitpod.io
GNU Affero General Public License v3.0
12.79k stars 1.23k forks source link

Enabling 'Cancel Prebuilds on Outdated Commits' Causes Some Prebuilds to Be Marked as SYSTEM ERROR / Failed #13874

Open jimmybrancaccio opened 1 year ago

jimmybrancaccio commented 1 year ago

Bug description

When enabling the 'Cancel Prebuilds on Outdated Commits' functionality on a project it can cause some prebuilds to go into a SYSTEM ERROR state and the prebuilt workspace is marked as failed. It also logs as an ERROR in the cluster - "workspace failed". This appears to mostly happen when a prebuild is longer running and a new commit is pushed to the projects repository.

While there actually does seem to be an error because the prebuild is "prematurely" stopped I don't believe we should mark this as an error or as failed. The reason why there's an error is because it was cancelled which can causes processes (like init tasks) to just exit or not run to their completion.

Screen Shot 2022-10-14 at 10 09 47

You'll notice in that screenshot the top prebuild is okay because I didn't make any more quick commit and pushes so the prebuild was allowed to fully run.

I believe there should be two things done here:

Steps to reproduce

  1. You can fork my repository I used to reproduce this issue.
  2. Setup a new project in Gitpod using this repository.
  3. Within the projects Settings > General, ensure that 'Cancel Prebuilds on Outdated Commits' is checked off.
  4. Open a new workspace for this project.
  5. You can use the test.txt file to make modifications too. I just kept adding a couple characters, or whatever and then committed and pushed the change.
  6. Modify the file again, commit and push the changes again. I repeated this 2-3 more times.
  7. Observe on the prebuild tab of your project that you see a bunch of SYSTEM ERRORs.

Workspace affected

jimmybranca-gitpoddocke-0y4i1f054dq

Expected behavior

I would expect that workspaces that are cancelled when using the 'Cancel Prebuilds on Outdated Commits' are left marked as Cancelled in the UI.

Example repository

https://github.com/jimmybrancaccio/gitpod-docker-compose

Anything else?

SELECT * FROM d_b_prebuilt_workspace WHERE id = 'a669c48a-2c25-4a3b-84bc-bb3b35b5b10d';

Front logo Front conversations

gtsiolis commented 1 year ago

Nice catch, @jimmybrancaccio! ⚾

Not sure if this is related, but the GitLab project is experiencing these system errors and the project has this project setting enabled, see relevant issue and relevant discussions[1][2], see relevant project (internal).