cirruslabs / gitlab-tart-executor

GitLab Runner executor to run jobs in Tart VMs
MIT License
60 stars 5 forks source link

A canceled GitLab CI job does not stop the tart runner VM #36

Closed rgl closed 1 year ago

rgl commented 1 year ago

When we cancel a GitLab CI job from the GitLab UI, the tart VM is not destroyed.

Accordingly to the GitLab documentation, GitLab will call the cleanup_exec to terminate/cleanup the runner when a job is canceled. Maybe this tart executor is not handling that part as a forcefully destroy the VM?

For reference, this is how gitlab tart runner is configured:

[[runners]]
  name = "tart"
  url = "https://git.example.com"
  id = 166
  token = "xxx"
  token_obtained_at = 2023-08-29T12:23:35Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "custom"
  builds_dir = "/builds"
  cache_dir = "/cache"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.custom]
    config_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    config_args = ["config"]
    config_exec_timeout = 120
    prepare_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    prepare_args = ["prepare", "--concurrency", "2", "--cpu", "auto", "--memory", "auto"]
    prepare_exec_timeout = 120
    run_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    run_args = ["run"]
    cleanup_exec = "/opt/homebrew/bin/gitlab-tart-executor"
    cleanup_args = ["cleanup"]
    cleanup_exec_timeout = 120
edigaryev commented 1 year ago

What does the job log say? Especially the last lines of it.

cleanup stage should normally report an error if it fails to stop the Tart VM.

rgl commented 1 year ago

Oh, I was also trying to run the gitlab runner as a Daemon instead of an Agent. And changing the Daemon setting from:

    <key>SessionCreate</key>
    <false/>

To:

    <key>SessionCreate</key>
    <true/>

Made cancelling work.

Sorry for the noise.