search

traderepublic / Cilicon

🛠️ Self-Hosted ephemeral macOS CI on Apple Silicon
MIT License
950 stars 28 forks source link

Cilicon runner don't restart after gitlab job is finished in Cilicon version 2.2.0 #52

Closed fberndtsson closed 3 weeks ago

fberndtsson commented 1 month ago

Issue:

Cilicon 2.2.0 vm don't restart when gitlab job is done as shown in GIF. Followed setup in Readme.md

Background:

From the cilion cycle the vm is stopped and removed after a job is done.

In the instructions it is not mentioned any setting required to restart Cilion VM. From the additional swift settings it seems numberOfRunsUntilHostReboot is the most likley setting to make it restart.

Settings

Cilicon version: v2.2.0

Mac version

ProductName:        macOS
ProductVersion:     14.5
BuildVersion:       23F79

cat ~/cilicon.yml

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: gitlab
  config:
    gitlabURL: https://gitlab.com
    runnerToken: <replace with gitlab-ci-cd-token>
    maxNumberOfBuilds: 1
numberOfRunsUntilHostReboot: 1

Running:

Configuring GitLab Runner...
Successfully configured GitLab Runner
Downloading GitLab Runner Binary from Source
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  3 61.5M    3 2330k    0     0  1944k      0  0:00:32  0:00:01  0:00:31 1943k
 26 61.5M   26 16.1M    0     0  7747k      0  0:00:08  0:00:02  0:00:06 7746k
 43 61.5M   43 26.5M    0     0  8614k      0  0:00:07  0:00:03  0:00:04 8613k
 60 61.5M   60 36.9M    0     0  9153k      0  0:00:06  0:00:04  0:00:02 9153k
 78 61.5M   78 48.0M    0     0  9578k      0  0:00:06  0:00:05  0:00:01 9989k
 96 61.5M   96 59.4M    0     0  9896k      0  0:00:06  0:00:06 --:--:-- 11.5M
100 61.5M  100 61.5M    0     0  9921k      0  0:00:06  0:00:06 --:--:-- 10.7M
Downloaded GitLab Runner Binary from Source successfully
Starting GitLab Runner...
Runtime platform                                    arch=arm64 os=darwin pid=660 revision=44feccdf version=17.0.0
Starting multi-runner from /Users/admin/.gitlab-runner/config.toml...  builds=0 max_builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...

. . .

Submitting job to coordinator...accepted, but not yet completed  bytesize=707934 checksum=checksum code=202 job=[job-number] job-status=running runner=[runner-id] update-interval=1s

Submitting job to coordinator...ok.      bytesize=707934 checksum=[checksum] code=200 job=[job-number] job-status=success runner=[runner-id] update-interval=0s

Removed job from processing list.        builds=0 job=[job-number] max_builds=1 project=[gitlab-project]  repo_url=[gitlab-repo-url]  time_in_queue_seconds=0
Marcocanc commented 1 month ago

@fberndtsson numberOfRunsUntilHostReboot reboots the machine running Cilicon after a certain number of runs (VM restarts). This is an optional feature to have a fresh boot every once in a while. Cilicon will restart its VM when the command that it runs via SSH exits. Since I don't use gitlab, I can't replicate your issue, but to me it looks like the command that is run never exits. This theory is supported by the fact that your log claims to be a "multi-runner" with max_builds=0. Maybe Gitlab added a multi-runner capability recently which would have broken the provisioner. It should be possible to fix this by changing run to run-single here. Would it be possible for you to test this? If you don't have XCode I can create a test build for you.

Marcocanc commented 1 month ago

@ast3150 Might know more about this.

fberndtsson commented 1 month ago

@Marcocanc

Thanks for the hint, it sounds like it could be it! I am a novice mac-developer, give me a few days and I will have tried out your suggestion.

Will post an update in a couple of days (or earlier) when I have tried it.

fberndtsson commented 1 month ago

Attempt 1 @Marcocanc suggestion:

gitlab-runner run-single

Made the virtual machine restart. No job was executed.

`Downloaded GitLab Runner Binary from Source successfully Starting GitLab Runner... Runtime platform  arch=arm64 os=darwin pid=774 revision=44feccdf version=17.0.0 FATAL: Missing URL  ---------- Shutting Down ---------- Configuring GitLab Runner... Successfully configured GitLab Runner Downloading GitLab Runner Binary from Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2 61.5M 2 1620k 0 0 1767k 0 0:00:35 --:--:-- 0:00:35 1767k 100 61.5M 100 61.5M 0 0 10.9M 0 0:00:05 0:00:05 --:--:-- 12.7M Downloaded GitLab Runner Binary from Source successfully Starting GitLab Runner... Runtime platform  `

Attempt 2

According to docs; gitlab-runner run-single -c ~/.gitlab-runner/config.toml -r runner-name should be used. MR

Tried with: let runCommand = "gitlab-runner run-single -c ~/.gitlab-runner/config.toml"

Made; gitlab-runner go crazy. With error message:

FATAL: flag provided but not defined: -c

before restart.

Currently trying to see that the file is existing within the vm. Then also, if something else needs to be modified to execute command with parameters.

fberndtsson commented 1 month ago

The MR: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/4789/diffs was merged 17th of May. Latest gitlab-runner released 16th of May, so the gitlab docs are incorrect as of now. Next version of gitlab-runner hopefully have the -c option released.

Will continue trying it to work when next version of gitlab-runner released.

Marcocanc commented 1 month ago

@fberndtsson if the issue is indeed in the latest version of GL Runner, you can manually specify the download URL in your config and set an older version.

fberndtsson commented 3 weeks ago

The issue is that we want to use the parameters from a file, and that feature is not included in the gitlab-runner as of now.

I want to see if gitlab enable it in the next version of the gitlab-runner. As it should be according to the documentation.

If the feature is missing in next version of gitlab-runner, I will see if we can get it running with the commandline-parameters. Unsure if there is any security implication of having the token on the command-line.

Marcocanc commented 3 weeks ago

@fberndtsson If I understand you correctly, this is an issue of the gitlab runner and not of Cilicon. I will therefore close the ticket. If I misunderstood, feel free to reopen it.

fberndtsson commented 3 weeks ago

Sorry for being a bit slow @Marcocanc . Have been testing out the patch the last week to ensure that it worked properly.

The paramter -c don't exists in gitlab-runner mode run-single even in 17.2.

Workaround is to send it via the cmd to use the gitlab-runner run-single -u -t --executor --maxbuild