search

cloudbase / garm

GitHub Actions Runner Manager
Apache License 2.0
138 stars 26 forks source link

garm-provider-gcp failing to leverage cached actions-runner software #257

Closed Hdom closed 5 months ago

Hdom commented 5 months ago

Provider: garm-provider-gcp

Issue: We tried to cache the actions-runner software at /opt/cache/actions-runner and it seems that there is a problem in the install-runner.sh that is causing an issue.

Setup:

mkdir -p /opt/cache/actions-runner/latest
cd /opt/cache/actions-runner/latest
version=$(git ls-remote --tags --refs --sort="version:refname" https://github.com/actions/runner.git | awk -F/ 'END{print$NF}')
version=${version#v}
curl -o actions-runner-linux-x64-$version.tar.gz -L https://github.com/actions/runner/releases/download/v$version/actions-runner-linux-x64-$version.tar.gz
tar xzf ./actions-runner-linux-x64-$version.tar.gz
./bin/installdependencies.sh
sudo chown -R runner:runner /opt/cache/actions-runner

Error:

| Status Updates  | 2024-06-06T17:27:02: using cached runner found in /opt/cache/actions-runner/latest |
|                 | 2024-06-06T17:27:08: configuring runner                                            |
|                 | 2024-06-06T17:27:08: downloading JIT credentials                                   |
|                 | 2024-06-06T17:27:09: generating systemd unit file                                  |
|                 | 2024-06-06T17:27:09: enabling runner service                                       |
|                 | 2024-06-06T17:27:09: failed to copy runsvc.sh

Potential Root Cause: https://github.com/cloudbase/garm-provider-common/blob/main/cloudconfig/templates.go#L129 This copying the latest folder, so result looks like this: /home/runner/actions-runner/latest

https://github.com/cloudbase/garm-provider-common/blob/main/cloudconfig/templates.go#L163 This is then trying to find /home/runner/actions-runner/bin/runsvc.sh but that file is now inside of /home/runner/actions-runner/latest/bin/runsvc.sh

Additional Thoughts: If you add /. to the cp command's source argument, like so: sudo cp -a "$CACHED_RUNNER/." "/home/{{ .RunnerUsername }}/actions-runner" it copies the content of latest and then the pathing is corrected.

gabriel-samfira commented 5 months ago

Thanks @Hdom ! Will push a fix for this in the morning. I completely forgot to test the cached runner case.

gabriel-samfira commented 5 months ago

Would you mind running the following command on one of the runners that currently fails in GCP:

ls -l /opt/cache/actions-runner/latest

I tried to reproduce with the LXD provider (almost all providers use the same userdata/script) and it seems to work as expected.

I will try the GCP provider as well, but I need to generate a custom image that includes the cached runner first.

gabriel-samfira commented 5 months ago

Also, could you share what base image you used for your custom image?

fabi200123 commented 5 months ago

Hello @Hdom! Just tested this out with a custom image that has "sourceImage": "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240519"

For me everything went without a problem:

+-----------------+------------------------------------------------------------------------------------+
| FIELD           | VALUE                                                                              |
+-----------------+------------------------------------------------------------------------------------+
| ID              | 962c4319-ee7d-41b0-bb79-ff1ce08ede9c                                               |
| Provider ID     | garm-nxj6em4u3auj                                                                  |
| Name            | garm-nxJ6EM4U3auj                                                                  |
| OS Type         | linux                                                                              |
| OS Architecture | amd64                                                                              |
| OS Name         | Ubuntu                                                                             |
| OS Version      | 20.04                                                                              |
| Status          | running                                                                            |
| Runner Status   | idle                                                                               |
| Pool ID         | 6d7c5507-722b-471c-af19-e4ca34afc241                                               |
| Status Updates  | 2024-06-07T08:17:41: using cached runner found in /opt/cache/actions-runner/latest |
|                 | 2024-06-07T08:18:05: configuring runner                                            |
|                 | 2024-06-07T08:18:05: downloading JIT credentials                                   |
|                 | 2024-06-07T08:18:06: generating systemd unit file                                  |
|                 | 2024-06-07T08:18:06: enabling runner service                                       |
|                 | 2024-06-07T08:18:07: runner successfully installed                                 |
+-----------------+------------------------------------------------------------------------------------+

And the runner ended up in idle:

+----+-------------------+---------+---------------+--------------------------------------+
| NR | NAME              | STATUS  | RUNNER STATUS | POOL ID                              |
+----+-------------------+---------+---------------+--------------------------------------+
|  1 | garm-nxJ6EM4U3auj | running | idle          | 6d7c5507-722b-471c-af19-e4ca34afc241 |
+----+-------------------+---------+---------------+--------------------------------------+

What image did you use that you had an issue?

Hdom commented 5 months ago

We use "projects/debian-cloud/global/images/family/debian-11" as the base image. The code for the cache setup I grabbed from https://github.com/cloudbase/garm/blob/main/doc/performance_considerations.md

@gabriel-samfira Here is the result of ls -l /opt/cache/actions-runner/latest

root@dho-test3:/home/debian# ls -l /opt/cache/actions-runner/latest
total 182512
-rw-r--r-- 1 runner runner 186845153 Jun  6 16:26 actions-runner-linux-x64-2.317.0.tar.gz
drwxr-xr-x 4 runner runner     16384 May 30 14:50 bin
-rwxr-xr-x 1 runner runner      2458 May 30 14:49 config.sh
-rwxr-xr-x 1 runner runner       646 May 30 14:49 env.sh
drwxr-xr-x 6 runner runner      4096 May 30 14:49 externals
-rw-r--r-- 1 runner runner      1619 May 30 14:49 run-helper.cmd.template
-rwxr-xr-x 1 runner runner      2663 May 30 14:49 run-helper.sh.template
-rwxr-xr-x 1 runner runner      2535 May 30 14:49 run.sh
-rwxr-xr-x 1 runner runner        65 May 30 14:49 safe_sleep.sh
fabi200123 commented 5 months ago

Just tested it out with the same base image and it worked okay:

Every 2.0s: garm-cli runner show garm-AZ198FzZSbC7                                                    garm-testing: Fri Jun  7 14:15:37 2024

+-----------------+------------------------------------------------------------------------------------+
| FIELD           | VALUE                                                                              |
+-----------------+------------------------------------------------------------------------------------+
| ID              | e5e988c5-95fc-4544-aa87-ec835cf4b5d8                                               |
| Provider ID     | garm-az198fzzsbc7                                                                  |
| Name            | garm-AZ198FzZSbC7                                                                  |
| OS Type         | linux                                                                              |
| OS Architecture | amd64                                                                              |
| OS Name         | Debian GNU/Linux                                                                   |
| OS Version      | 11                                                                                 |
| Status          | running                                                                            |
| Runner Status   | idle                                                                               |
| Pool ID         | 6d7c5507-722b-471c-af19-e4ca34afc241                                               |
| Status Updates  | 2024-06-07T14:15:13: using cached runner found in /opt/cache/actions-runner/latest |
|                 | 2024-06-07T14:15:31: configuring runner                                            |
|                 | 2024-06-07T14:15:31: downloading JIT credentials                                   |
|                 | 2024-06-07T14:15:31: generating systemd unit file                                  |
|                 | 2024-06-07T14:15:31: enabling runner service                                       |
|                 | 2024-06-07T14:15:32: runner successfully installed                                 |
+-----------------+------------------------------------------------------------------------------------+
garm-cli pool show 6d7c5507-722b-471c-af19-e4ca34afc241
+--------------------------+--------------------------------------------------------------+
| FIELD                    | VALUE                                                        |
+--------------------------+--------------------------------------------------------------+
| ID                       | 6d7c5507-722b-471c-af19-e4ca34afc241                         |
| Provider Name            | gcp                                                          |
| Image                    | projects/garm-testing-424210/global/images/debian-garm-2-317 |

In the garm instance it created, everything is copied okay from the /opt/cache/actions-runner/latest/:

root@garm-ocwp2ygeqywa:/home/runner/actions-runner# ls -l /opt/cache/actions-runner/latest/
total 182512
-rw-r--r-- 1 runner runner 186845153 Jun  7 14:08 actions-runner-linux-x64-2.317.0.tar.gz
drwxr-xr-x 4 runner runner     16384 May 30 14:50 bin
-rwxr-xr-x 1 runner runner      2458 May 30 14:49 config.sh
-rwxr-xr-x 1 runner runner       646 May 30 14:49 env.sh
drwxr-xr-x 6 runner runner      4096 May 30 14:49 externals
-rw-r--r-- 1 runner runner      1619 May 30 14:49 run-helper.cmd.template
-rwxr-xr-x 1 runner runner      2663 May 30 14:49 run-helper.sh.template
-rwxr-xr-x 1 runner runner      2535 May 30 14:49 run.sh
-rwxr-xr-x 1 runner runner        65 May 30 14:49 safe_sleep.sh
root@garm-ocwp2ygeqywa:/home/runner/actions-runner# ls -l
total 182520
drwxr-xr-x 2 runner runner      4096 Jun  7 14:26 _diag
-rw-r--r-- 1 runner runner 186845153 Jun  7 14:08 actions-runner-linux-x64-2.317.0.tar.gz
drwxr-xr-x 4 runner runner     16384 May 30 14:50 bin
-rwxr-xr-x 1 runner runner      2458 May 30 14:49 config.sh
-rwxr-xr-x 1 runner runner       646 May 30 14:49 env.sh
drwxr-xr-x 6 runner runner      4096 May 30 14:49 externals
-rw-r--r-- 1 runner runner      1619 May 30 14:49 run-helper.cmd.template
-rwxr-xr-x 1 runner runner      2663 May 30 14:49 run-helper.sh.template
-rwxr-xr-x 1 runner runner      2535 May 30 14:49 run.sh
-rwxr-xr-x 1 runner runner       576 Jun  7 14:26 runsvc.sh
-rwxr-xr-x 1 runner runner        65 May 30 14:49 safe_sleep.sh
Hdom commented 5 months ago

I was hoping it could be explained by a debian vs ubuntu diff.

Here is our runner:

$ gc runner show gtd4vlmmnd-hwvm1cWdtoZp
+-----------------+------------------------------------------------------------------------------------+
| FIELD           | VALUE                                                                              |
+-----------------+------------------------------------------------------------------------------------+
| ID              | 65af16c2-de17-42d4-9f39-240a663aa3bc                                               |
| Provider ID     | gtd4vlmmnd-hwvm1cWdtoZp                                                            |
| Name            | gtd4vlmmnd-hwvm1cWdtoZp                                                            |
| OS Type         | linux                                                                              |
| OS Architecture | amd64                                                                              |
| OS Name         |                                                                                    |
| OS Version      |                                                                                    |
| Status          | running                                                                            |
| Runner Status   | failed                                                                             |
| Pool ID         | f385432a-b789-4920-b7b9-b0dcd22549fe                                               |
| Status Updates  | 2024-06-07T14:35:41: using cached runner found in /opt/cache/actions-runner/latest |
|                 | 2024-06-07T14:35:48: configuring runner                                            |
|                 | 2024-06-07T14:35:48: downloading JIT credentials                                   |
|                 | 2024-06-07T14:35:48: generating systemd unit file                                  |
|                 | 2024-06-07T14:35:48: enabling runner service                                       |
|                 | 2024-06-07T14:35:48: failed to copy runsvc.sh                                      |
+-----------------+------------------------------------------------------------------------------------+

Here is a look inside the runner:

root@gtd4vlmmnd-6877766d31635764746f5a70:/home/debian# ls -l /opt/cache/actions-runner/latest/
total 182512
-rw-r--r-- 1 runner runner 186845153 Jun  6 16:26 actions-runner-linux-x64-2.317.0.tar.gz
drwxr-xr-x 4 runner runner     16384 May 30 14:50 bin
-rwxr-xr-x 1 runner runner      2458 May 30 14:49 config.sh
-rwxr-xr-x 1 runner runner       646 May 30 14:49 env.sh
drwxr-xr-x 6 runner runner      4096 May 30 14:49 externals
-rw-r--r-- 1 runner runner      1619 May 30 14:49 run-helper.cmd.template
-rwxr-xr-x 1 runner runner      2663 May 30 14:49 run-helper.sh.template
-rwxr-xr-x 1 runner runner      2535 May 30 14:49 run.sh
-rwxr-xr-x 1 runner runner        65 May 30 14:49 safe_sleep.sh
root@gtd4vlmmnd-6877766d31635764746f5a70:/home/debian# ls -l /home/runner/actions-runner
total 4
drwxr-xr-x 4 runner runner 4096 May 30 14:50 latest

Maybe we are using an old version of common? Have there been any changes to the install-runner.sh template recently?

gabriel-samfira commented 5 months ago

Could be. Could you vlone the latest main of the gcp provider and try that out?

Hdom commented 5 months ago

Updated to latest, which doesn't seem to have updated common past v0.1.2 and there is no difference.

You can see in the check inside of /home/runner/actions-runner folder that the copy brought over the latest folder instead of its content. 

root@gtd4vlmmnd-6877766d31635764746f5a70:/home/debian# ls -l /home/runner/actions-runner
total 4
drwxr-xr-x 4 runner runner 4096 May 30 14:50 latest

And the code in the startup-script looks like so:

function getCachedToolsPath() {
    CACHED_RUNNER="/opt/cache/actions-runner/latest"
    if [ -d "$CACHED_RUNNER" ];then
        echo "$CACHED_RUNNER"
        return 0
    fi

    VERSION=$(getRunnerVersion)
    if [ -z "$VERSION" ]; then
        return 0
    fi

    CACHED_RUNNER="/opt/cache/actions-runner/$VERSION"
    if [ -d "$CACHED_RUNNER" ];then
        echo "$CACHED_RUNNER"
        return 0
    fi
    return 0
}
...
CACHED_RUNNER=$(getCachedToolsPath)
if [ -z "$CACHED_RUNNER" ];then
    downloadAndExtractRunner
    sendStatus "installing dependencies"
    cd /home/runner/actions-runner
    sudo ./bin/installdependencies.sh || fail "failed to install dependencies"
else
    sendStatus "using cached runner found in $CACHED_RUNNER"
    sudo cp -a "$CACHED_RUNNER"  "/home/runner/actions-runner"
    sudo chown runner:runner -R "/home/runner/actions-runner" || fail "failed to change owner"
    cd /home/runner/actions-runner
fi

I am out of ideas. 😓 I will continue tinkering with it and let you know if I have any breakthroughs, thank you for your time.

gabriel-samfira commented 5 months ago

Hmm. What do you have in the bin folder of the cached runner?

Hdom commented 5 months ago 
root@gtd4vlmmnd-704334684c796e704c77484d:/home/debian# ls -l /opt/cache/actions-runner/latest/bin
total 77636
-rwxr--r-- 1 runner runner   392736 Nov 10  2023 Azure.Core.dll
-rwxr--r-- 1 runner runner  1308080 Nov 13  2023 Azure.Storage.Blobs.dll
-rwxr--r-- 1 runner runner   243632 Nov 13  2023 Azure.Storage.Common.dll
-rwxr--r-- 1 runner runner    14720 Apr 14  2020 Microsoft.Bcl.AsyncInterfaces.dll
-rwxr--r-- 1 runner runner   841888 Mar 21 17:29 Microsoft.CSharp.dll
-rwxr--r-- 1 runner runner    30832 Feb  8  2018 Microsoft.IdentityModel.Logging.dll
-rwxr--r-- 1 runner runner   138352 Feb  8  2018 Microsoft.IdentityModel.Tokens.dll
-rwxr--r-- 1 runner runner  1233056 Mar 21 17:29 Microsoft.VisualBasic.Core.dll
-rwxr--r-- 1 runner runner    18096 Mar 21 17:08 Microsoft.VisualBasic.dll
-rwxr--r-- 1 runner runner    24752 Mar 21 17:29 Microsoft.Win32.Primitives.dll
-rwxr--r-- 1 runner runner    59040 Mar 21 17:29 Microsoft.Win32.Registry.dll
-rwxr--r-- 1 runner runner    18432 Jun  5  2016 Minimatch.dll
-rwxr--r-- 1 runner runner    90624 Mar 22  2017 Newtonsoft.Json.Bson.dll
-rwxr--r-- 1 runner runner   712464 Mar  8  2023 Newtonsoft.Json.dll
-rw-r--r-- 1 runner runner    92086 May 30 14:49 Runner.Common.deps.json
-rw-r--r-- 1 runner runner   195584 May 30 14:49 Runner.Common.dll
-rwxr-xr-x 1 runner runner   142840 May 30 14:49 Runner.Listener
-rw-r--r-- 1 runner runner   159283 May 30 14:49 Runner.Listener.deps.json
-rw-r--r-- 1 runner runner   232960 May 30 14:49 Runner.Listener.dll
-rw-r--r-- 1 runner runner      332 May 30 14:49 Runner.Listener.runtimeconfig.json
-rwxr-xr-x 1 runner runner   142840 May 30 14:49 Runner.PluginHost
-rw-r--r-- 1 runner runner   157639 May 30 14:49 Runner.PluginHost.deps.json
-rw-r--r-- 1 runner runner     6656 May 30 14:49 Runner.PluginHost.dll
-rw-r--r-- 1 runner runner      332 May 30 14:49 Runner.PluginHost.runtimeconfig.json
-rw-r--r-- 1 runner runner    91837 May 30 14:49 Runner.Plugins.deps.json
-rw-r--r-- 1 runner runner   111616 May 30 14:49 Runner.Plugins.dll
-rw-r--r-- 1 runner runner    91526 May 30 14:49 Runner.Sdk.deps.json
-rw-r--r-- 1 runner runner    53248 May 30 14:49 Runner.Sdk.dll
-rwxr-xr-x 1 runner runner   142840 May 30 14:49 Runner.Worker
-rw-r--r-- 1 runner runner   158612 May 30 14:49 Runner.Worker.deps.json
-rw-r--r-- 1 runner runner   364544 May 30 14:49 Runner.Worker.dll
-rw-r--r-- 1 runner runner      332 May 30 14:49 Runner.Worker.runtimeconfig.json
-rw-r--r-- 1 runner runner     4832 May 30 14:49 RunnerService.js
-rw-r--r-- 1 runner runner    90252 May 30 14:49 Sdk.deps.json
-rw-r--r-- 1 runner runner  1089024 May 30 14:49 Sdk.dll
-rwxr--r-- 1 runner runner    15640 Mar 21 17:08 System.AppContext.dll
-rwxr--r-- 1 runner runner    15520 Mar 21 17:08 System.Buffers.dll
-rwxr--r-- 1 runner runner   238768 Mar 21 17:29 System.Collections.Concurrent.dll
-rwxr--r-- 1 runner runner   600224 Mar 21 17:29 System.Collections.Immutable.dll
-rwxr--r-- 1 runner runner   101128 Mar 21 17:29 System.Collections.NonGeneric.dll
-rwxr--r-- 1 runner runner    95920 Mar 21 17:29 System.Collections.Specialized.dll
-rwxr--r-- 1 runner runner   268960 Mar 21 17:29 System.Collections.dll
-rwxr--r-- 1 runner runner   187552 Mar 21 17:29 System.ComponentModel.Annotations.dll
-rwxr--r-- 1 runner runner    17688 Mar 21 17:08 System.ComponentModel.DataAnnotations.dll
-rwxr--r-- 1 runner runner    38560 Mar 21 17:29 System.ComponentModel.EventBasedAsync.dll
-rwxr--r-- 1 runner runner    75936 Mar 21 17:29 System.ComponentModel.Primitives.dll
-rwxr--r-- 1 runner runner   752288 Mar 21 17:29 System.ComponentModel.TypeConverter.dll
-rwxr--r-- 1 runner runner    18592 Mar 21 17:29 System.ComponentModel.dll
-rwxr--r-- 1 runner runner    19616 Mar 21 17:08 System.Configuration.dll
-rwxr--r-- 1 runner runner   190640 Mar 21 17:29 System.Console.dll
-rwxr--r-- 1 runner runner    24224 Mar 21 17:08 System.Core.dll
-rwxr--r-- 1 runner runner  3017392 Mar 21 17:29 System.Data.Common.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Data.DataSetExtensions.dll
-rwxr--r-- 1 runner runner    25864 Mar 21 17:08 System.Data.dll
-rwxr--r-- 1 runner runner    16152 Mar 21 17:08 System.Diagnostics.Contracts.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Diagnostics.Debug.dll
-rwxr--r-- 1 runner runner   384672 Mar 21 17:30 System.Diagnostics.DiagnosticSource.dll
-rwxr--r-- 1 runner runner    41632 Mar 21 17:30 System.Diagnostics.FileVersionInfo.dll
-rwxr--r-- 1 runner runner   264880 Mar 21 17:30 System.Diagnostics.Process.dll
-rwxr--r-- 1 runner runner    36624 Mar 21 17:30 System.Diagnostics.StackTrace.dll
-rwxr--r-- 1 runner runner    60592 Mar 21 17:30 System.Diagnostics.TextWriterTraceListener.dll
-rwxr--r-- 1 runner runner    16152 Mar 21 17:08 System.Diagnostics.Tools.dll
-rwxr--r-- 1 runner runner   133296 Mar 21 17:30 System.Diagnostics.TraceSource.dll
-rwxr--r-- 1 runner runner    16648 Mar 21 17:08 System.Diagnostics.Tracing.dll
-rwxr--r-- 1 runner runner   130224 Mar 21 17:30 System.Drawing.Primitives.dll
-rwxr--r-- 1 runner runner    21256 Mar 21 17:08 System.Drawing.dll
-rwxr--r-- 1 runner runner    16544 Mar 21 17:08 System.Dynamic.Runtime.dll
-rwxr--r-- 1 runner runner   203936 Mar 21 17:30 System.Formats.Asn1.dll
-rwxr--r-- 1 runner runner    16144 Mar 21 17:08 System.Globalization.Calendars.dll
-rwxr--r-- 1 runner runner    15536 Mar 21 17:08 System.Globalization.Extensions.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Globalization.dll
-rwxr--r-- 1 runner runner    73392 Mar 21 17:30 System.IO.Compression.Brotli.dll
-rwxr--r-- 1 runner runner    16032 Mar 21 17:08 System.IO.Compression.FileSystem.dll
-rwxr--r-- 1 runner runner    53408 Mar 21 17:30 System.IO.Compression.ZipFile.dll
-rwxr--r-- 1 runner runner   271008 Mar 21 17:30 System.IO.Compression.dll
-rwxr--r-- 1 runner runner    29872 Mar 21 17:30 System.IO.FileSystem.AccessControl.dll
-rwxr--r-- 1 runner runner    78096 Mar 21 17:30 System.IO.FileSystem.DriveInfo.dll
-rwxr--r-- 1 runner runner    15520 Mar 21 17:08 System.IO.FileSystem.Primitives.dll
-rwxr--r-- 1 runner runner    97440 Mar 21 17:30 System.IO.FileSystem.Watcher.dll
-rwxr--r-- 1 runner runner    24344 Mar 21 17:08 System.IO.FileSystem.dll
-rwxr--r-- 1 runner runner    29296 Oct 22  2021 System.IO.Hashing.dll
-rwxr--r-- 1 runner runner    82608 Mar 21 17:30 System.IO.IsolatedStorage.dll
-rwxr--r-- 1 runner runner    75936 Mar 21 17:30 System.IO.MemoryMappedFiles.dll
-rwxr--r-- 1 runner runner    24736 Mar 21 17:30 System.IO.Pipes.AccessControl.dll
-rwxr--r-- 1 runner runner   126112 Mar 21 17:30 System.IO.Pipes.dll
-rwxr--r-- 1 runner runner    15624 Mar 21 17:08 System.IO.UnmanagedMemoryStream.dll
-rwxr--r-- 1 runner runner    16152 Mar 21 17:08 System.IO.dll
-rwxr--r-- 1 runner runner    90736 Feb  8  2018 System.IdentityModel.Tokens.Jwt.dll
-rwxr--r-- 1 runner runner  3901600 Mar 21 17:30 System.Linq.Expressions.dll
-rwxr--r-- 1 runner runner   852128 Mar 21 17:30 System.Linq.Parallel.dll
-rwxr--r-- 1 runner runner   232096 Mar 21 17:30 System.Linq.Queryable.dll
-rwxr--r-- 1 runner runner   539808 Mar 21 17:30 System.Linq.dll
-rwxr--r-- 1 runner runner    21368 Apr  8  2021 System.Memory.Data.dll
-rwxr--r-- 1 runner runner   177312 Mar 21 17:30 System.Memory.dll
-rwxr--r-- 1 runner runner   179672 May 27  2022 System.Net.Http.Formatting.dll
-rwxr--r-- 1 runner runner    84656 Mar 21 17:30 System.Net.Http.Json.dll
-rwxr--r-- 1 runner runner  1774752 Mar 21 17:30 System.Net.Http.dll
-rwxr--r-- 1 runner runner   309920 Mar 21 17:30 System.Net.HttpListener.dll
-rwxr--r-- 1 runner runner   495776 Mar 21 17:30 System.Net.Mail.dll
-rwxr--r-- 1 runner runner    87216 Mar 21 17:30 System.Net.NameResolution.dll
-rwxr--r-- 1 runner runner   166560 Mar 21 17:30 System.Net.NetworkInformation.dll
-rwxr--r-- 1 runner runner   102048 Mar 21 17:30 System.Net.Ping.dll
-rwxr--r-- 1 runner runner   229040 Mar 21 17:30 System.Net.Primitives.dll
-rwxr--r-- 1 runner runner   299680 Mar 21 17:30 System.Net.Quic.dll
-rwxr--r-- 1 runner runner   351392 Mar 21 17:30 System.Net.Requests.dll
-rwxr--r-- 1 runner runner   703136 Mar 21 17:30 System.Net.Security.dll
-rwxr--r-- 1 runner runner    36512 Mar 21 17:30 System.Net.ServicePoint.dll
-rwxr--r-- 1 runner runner   582816 Mar 21 17:30 System.Net.Sockets.dll
-rwxr--r-- 1 runner runner   168624 Mar 21 17:30 System.Net.WebClient.dll
-rwxr--r-- 1 runner runner    61088 Mar 21 17:30 System.Net.WebHeaderCollection.dll
-rwxr--r-- 1 runner runner    31904 Mar 21 17:30 System.Net.WebProxy.dll
-rwxr--r-- 1 runner runner    76960 Mar 21 17:30 System.Net.WebSockets.Client.dll
-rwxr--r-- 1 runner runner   187040 Mar 21 17:30 System.Net.WebSockets.dll
-rwxr--r-- 1 runner runner    18080 Mar 21 17:08 System.Net.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Numerics.Vectors.dll
-rwxr--r-- 1 runner runner    16032 Mar 21 17:08 System.Numerics.dll
-rwxr--r-- 1 runner runner    91312 Mar 21 17:30 System.ObjectModel.dll
-rwxr--r-- 1 runner runner 10424496 Mar 21 17:14 System.Private.CoreLib.dll
-rwxr--r-- 1 runner runner  2098952 Mar 21 17:30 System.Private.DataContractSerialization.dll
-rwxr--r-- 1 runner runner   256792 Mar 21 17:30 System.Private.Uri.dll
-rwxr--r-- 1 runner runner   414896 Mar 21 17:30 System.Private.Xml.Linq.dll
-rwxr--r-- 1 runner runner  8687776 Mar 21 17:30 System.Private.Xml.dll
-rwxr--r-- 1 runner runner    66720 Mar 21 17:30 System.Reflection.DispatchProxy.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Reflection.Emit.ILGeneration.dll
-rwxr--r-- 1 runner runner    16144 Mar 21 17:08 System.Reflection.Emit.Lightweight.dll
-rwxr--r-- 1 runner runner    16160 Mar 21 17:08 System.Reflection.Emit.dll
-rwxr--r-- 1 runner runner    15632 Mar 21 17:08 System.Reflection.Extensions.dll
-rwxr--r-- 1 runner runner  1115296 Mar 21 17:30 System.Reflection.Metadata.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Reflection.Primitives.dll
-rwxr--r-- 1 runner runner    33440 Mar 21 17:30 System.Reflection.TypeExtensions.dll
-rwxr--r-- 1 runner runner    16664 Mar 21 17:08 System.Reflection.dll
-rwxr--r-- 1 runner runner    15640 Mar 21 17:08 System.Resources.Reader.dll
-rwxr--r-- 1 runner runner    16160 Mar 21 17:08 System.Resources.ResourceManager.dll
-rwxr--r-- 1 runner runner    45728 Mar 21 17:30 System.Resources.Writer.dll
-rwxr--r-- 1 runner runner    21784 Mar 21 17:30 System.Runtime.CompilerServices.Unsafe.dll
-rwxr--r-- 1 runner runner    20128 Mar 21 17:30 System.Runtime.CompilerServices.VisualC.dll
-rwxr--r-- 1 runner runner    18200 Mar 21 17:08 System.Runtime.Extensions.dll
-rwxr--r-- 1 runner runner    15520 Mar 21 17:08 System.Runtime.Handles.dll
-rwxr--r-- 1 runner runner    29872 Mar 21 17:30 System.Runtime.InteropServices.RuntimeInformation.dll
-rwxr--r-- 1 runner runner    52384 Mar 21 17:30 System.Runtime.InteropServices.dll
-rwxr--r-- 1 runner runner    16648 Mar 21 17:08 System.Runtime.Intrinsics.dll
-rwxr--r-- 1 runner runner    16160 Mar 21 17:08 System.Runtime.Loader.dll
-rwxr--r-- 1 runner runner   219296 Mar 21 17:30 System.Runtime.Numerics.dll
-rwxr--r-- 1 runner runner   325896 Mar 21 17:30 System.Runtime.Serialization.Formatters.dll
-rwxr--r-- 1 runner runner    16032 Mar 21 17:08 System.Runtime.Serialization.Json.dll
-rwxr--r-- 1 runner runner    28936 Mar 21 17:30 System.Runtime.Serialization.Primitives.dll
-rwxr--r-- 1 runner runner    16648 Mar 21 17:08 System.Runtime.Serialization.Xml.dll
-rwxr--r-- 1 runner runner    17696 Mar 21 17:08 System.Runtime.Serialization.dll
-rwxr--r-- 1 runner runner    42760 Mar 21 17:08 System.Runtime.dll
-rwxr--r-- 1 runner runner    53528 Mar 21 17:30 System.Security.AccessControl.dll
-rwxr--r-- 1 runner runner    94472 Mar 21 17:30 System.Security.Claims.dll
-rwxr--r-- 1 runner runner   757408 Mar 21 17:30 System.Security.Cryptography.Algorithms.dll
-rwxr--r-- 1 runner runner    49424 Mar 21 17:31 System.Security.Cryptography.Cng.dll
-rwxr--r-- 1 runner runner   122632 Mar 21 17:31 System.Security.Cryptography.Csp.dll
-rwxr--r-- 1 runner runner   102160 Mar 21 17:31 System.Security.Cryptography.Encoding.dll
-rwxr--r-- 1 runner runner   210592 Mar 21 17:31 System.Security.Cryptography.OpenSsl.dll
-rwxr--r-- 1 runner runner    35080 Jul 19  2017 System.Security.Cryptography.Pkcs.dll
-rwxr--r-- 1 runner runner   137888 Mar 21 17:31 System.Security.Cryptography.Primitives.dll
-rwxr--r-- 1 runner runner    24880 Jul 19  2017 System.Security.Cryptography.ProtectedData.dll
-rwxr--r-- 1 runner runner   800928 Mar 21 17:31 System.Security.Cryptography.X509Certificates.dll
-rwxr--r-- 1 runner runner    36624 Mar 21 17:31 System.Security.Principal.Windows.dll
-rwxr--r-- 1 runner runner    15624 Mar 21 17:08 System.Security.Principal.dll
-rwxr--r-- 1 runner runner    15624 Mar 21 17:08 System.Security.SecureString.dll
-rwxr--r-- 1 runner runner    18592 Mar 21 17:08 System.Security.dll
-rwxr--r-- 1 runner runner    17688 Mar 21 17:08 System.ServiceModel.Web.dll
-rwxr--r-- 1 runner runner    27944 Jul 19  2017 System.ServiceProcess.ServiceController.dll
-rwxr--r-- 1 runner runner    16656 Mar 21 17:08 System.ServiceProcess.dll
-rwxr--r-- 1 runner runner   873632 Mar 21 17:31 System.Text.Encoding.CodePages.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Text.Encoding.Extensions.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Text.Encoding.dll
-rwxr--r-- 1 runner runner   130832 Mar 21 17:31 System.Text.Encodings.Web.dll
-rwxr--r-- 1 runner runner  1493784 Mar 21 17:31 System.Text.Json.dll
-rwxr--r-- 1 runner runner   537880 Mar 21 17:31 System.Text.RegularExpressions.dll
-rwxr--r-- 1 runner runner   126216 Mar 21 17:31 System.Threading.Channels.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Threading.Overlapped.dll
-rwxr--r-- 1 runner runner   509192 Mar 21 17:31 System.Threading.Tasks.Dataflow.dll
-rwxr--r-- 1 runner runner    16032 Mar 21 17:08 System.Threading.Tasks.Extensions.dll
-rwxr--r-- 1 runner runner   139424 Mar 21 17:31 System.Threading.Tasks.Parallel.dll
-rwxr--r-- 1 runner runner    17160 Mar 21 17:08 System.Threading.Tasks.dll
-rwxr--r-- 1 runner runner    16160 Mar 21 17:08 System.Threading.Thread.dll
-rwxr--r-- 1 runner runner    16144 Mar 21 17:08 System.Threading.ThreadPool.dll
-rwxr--r-- 1 runner runner    15640 Mar 21 17:08 System.Threading.Timer.dll
-rwxr--r-- 1 runner runner    80656 Mar 21 17:31 System.Threading.dll
-rwxr--r-- 1 runner runner   350368 Mar 21 17:31 System.Transactions.Local.dll
-rwxr--r-- 1 runner runner    17176 Mar 21 17:08 System.Transactions.dll
-rwxr--r-- 1 runner runner    15624 Mar 21 17:08 System.ValueTuple.dll
-rwxr--r-- 1 runner runner    53512 Mar 21 17:31 System.Web.HttpUtility.dll
-rwxr--r-- 1 runner runner    16152 Mar 21 17:08 System.Web.dll
-rwxr--r-- 1 runner runner    16544 Mar 21 17:08 System.Windows.dll
-rwxr--r-- 1 runner runner    16544 Mar 21 17:08 System.Xml.Linq.dll
-rwxr--r-- 1 runner runner    22176 Mar 21 17:08 System.Xml.ReaderWriter.dll
-rwxr--r-- 1 runner runner    16648 Mar 21 17:08 System.Xml.Serialization.dll
-rwxr--r-- 1 runner runner    16136 Mar 21 17:08 System.Xml.XDocument.dll
-rwxr--r-- 1 runner runner    18592 Mar 21 17:31 System.Xml.XPath.XDocument.dll
-rwxr--r-- 1 runner runner    16032 Mar 21 17:08 System.Xml.XPath.dll
-rwxr--r-- 1 runner runner    16048 Mar 21 17:08 System.Xml.XmlDocument.dll
-rwxr--r-- 1 runner runner    18184 Mar 21 17:08 System.Xml.XmlSerializer.dll
-rwxr--r-- 1 runner runner    24840 Mar 21 17:08 System.Xml.dll
-rwxr--r-- 1 runner runner    50968 Mar 21 17:08 System.dll
-rwxr--r-- 1 runner runner    17160 Mar 21 17:08 WindowsBase.dll
-rwxr--r-- 1 runner runner   184320 Dec  5  2018 YamlDotNet.dll
-rw-r--r-- 1 runner runner      940 May 30 14:49 actions.runner.plist.template
-rw-r--r-- 1 runner runner      242 May 30 14:49 actions.runner.service.template
drwxr-xr-x 2 runner runner     4096 May 30 14:49 checkScripts
-rwxr--r-- 1 runner runner   145856 Mar 21 17:07 createdump
-rw-r--r-- 1 runner runner     3535 May 30 14:49 darwin.svc.sh.template
drwxr-xr-x 2 runner runner     4096 May 30 14:49 hashFiles
-rwxr-xr-x 1 runner runner     7324 May 30 14:49 installdependencies.sh
-rwxr--r-- 1 runner runner    72104 Mar 21 17:08 libSystem.Globalization.Native.so
-rwxr--r-- 1 runner runner   940184 Mar 21 17:08 libSystem.IO.Compression.Native.so
-rwxr--r-- 1 runner runner    92272 Mar 21 17:08 libSystem.Native.so
-rwxr--r-- 1 runner runner    14472 Mar 21 17:08 libSystem.Net.Security.Native.so
-rwxr--r-- 1 runner runner   141456 Mar 21 17:08 libSystem.Security.Cryptography.Native.OpenSsl.so
-rwxr--r-- 1 runner runner  3359640 Mar 21 17:07 libclrjit.so
-rwxr--r-- 1 runner runner  7360032 Mar 21 17:07 libcoreclr.so
-rwxr--r-- 1 runner runner   747936 Mar 21 16:58 libcoreclrtraceptprovider.so
-rwxr--r-- 1 runner runner   738056 Mar 21 17:04 libdbgshim.so
-rwxr--r-- 1 runner runner   424280 Mar 21 17:25 libhostfxr.so
-rwxr--r-- 1 runner runner   383328 Mar 21 17:26 libhostpolicy.so
-rwxr--r-- 1 runner runner  2602272 Mar 21 17:07 libmscordaccore.so
-rwxr--r-- 1 runner runner  1778936 Mar 21 17:07 libmscordbi.so
-rw-r--r-- 1 runner runner      431 May 30 14:49 macos-run-invoker.js
-rwxr--r-- 1 runner runner    58632 Mar 21 17:08 mscorlib.dll
-rwxr--r-- 1 runner runner   101664 Mar 21 17:08 netstandard.dll
-rwxr-xr-x 1 runner runner      576 May 30 14:49 runsvc.sh
-rw-r--r-- 1 runner runner     5151 May 30 14:49 systemd.svc.sh.template
-rw-r--r-- 1 runner runner     5937 May 30 14:49 update.cmd.template
-rwxr-xr-x 1 runner runner     8748 May 30 14:49 update.sh.template
gabriel-samfira commented 5 months ago

Could you set the 

{
    "enable_boot_debug": true
}

Extra specs and create a runner? This will set the -x flag in the userdata script and we can see a trace of what is running. Maybe we can get an idea where it fails

fabi200123 commented 5 months ago

I think that there might be an issue with the way you set up your custom-image, what is the code you used to set up your image?

Hdom commented 5 months ago

@fabi200123 The code that adds the cache is the following:

mkdir -p /opt/cache/actions-runner/latest
cd /opt/cache/actions-runner/latest
version=$(git ls-remote --tags --refs --sort="version:refname" https://github.com/actions/runner.git | awk -F/ 'END{print$NF}')
version=${version#v}
curl -o actions-runner-linux-x64-$version.tar.gz -L https://github.com/actions/runner/releases/download/v$version/actions-runner-linux-x64-$version.tar.gz
tar xzf ./actions-runner-linux-x64-$version.tar.gz
./bin/installdependencies.sh
sudo chown -R runner:runner /opt/cache/actions-runner

@gabriel-samfira setting it up now

gabriel-samfira commented 5 months ago

@Hdom Recently we started enforcing the jsonschema. We forgot to add the common extra specs provided by the garm-provider-common package. I added them now in this PR:

https://github.com/cloudbase/garm-provider-gcp/pull/15

You might need that for enable_boot_debug to work. Would you be willing to give that a shot? Also, the PR adds a runner_install_template and extra_context extra specs.

These two options together allows you to override the entire runner install template, for easy testing. If you grab the entire template from:

https://github.com/cloudbase/garm-provider-common/blob/d0fe67934a5bcb773503553555274080ba60a852/cloudconfig/templates.go#L26-L228

Convert it to base64 and set it in extra_specs as:

{
    "runner_install_template": "base64GoesHere"
}

It will be used instead of the default internal one. All the same context is sent to it. So the same variables you see in the default template, will be available in your custom template as well.

Additionally, if you need to inject extra context beside what is already being sent, you can use the extra_context extra spec which translates to this variable:

https://github.com/cloudbase/garm-provider-common/blob/d0fe67934a5bcb773503553555274080ba60a852/cloudconfig/templates.go#L567

a string key/value pair which can be defined like:

{
    "extra_context": {
        "Person" : "Hdom"
    }
}

And in the template can be expanded using: {{ .ExtraContext.Person }}

Hdom commented 5 months ago

@gabriel-samfira sounds good I will give it a try once its merged

gabriel-samfira commented 5 months ago

merged

gabriel-samfira commented 5 months ago

Here is an example of extra_specs that overrides the default runner install template:

{
  "enable_boot_debug": true,
  "extra_context": {
    "Person": "Gabriel",
    "Who": "world"
  },
  "runner_install_template": "IyEvYmluL2Jhc2gKCnNldCAtZQpzZXQgLW8gcGlwZWZhaWwKCnt7LSBpZiAuRW5hYmxlQm9vdERlYnVnIH19CnNldCAteAp7ey0gZW5kIH19CgplY2hvICJJIGFtIHRoZSBvdmVycmlkZSBzY3JpcHQ6IHt7IC5FeHRyYUNvbnRleHQuUGVyc29uIH19IC0ge3sgLkV4dHJhQ29udGV4dC5XaG8gfX0iIHwgc3VkbyB0ZWUgL2dhcm0tb3ZlcnJpZGUudHh0CgpDQUxMQkFDS19VUkw9Int7IC5DYWxsYmFja1VSTCB9fSIKTUVUQURBVEFfVVJMPSJ7eyAuTWV0YWRhdGFVUkwgfX0iCkJFQVJFUl9UT0tFTj0ie3sgLkNhbGxiYWNrVG9rZW4gfX0iCgppZiBbIC16ICIkTUVUQURBVEFfVVJMIiBdO3RoZW4KCWVjaG8gIm5vIHRva2VuIGlzIGF2YWlsYWJsZSBhbmQgTUVUQURBVEFfVVJMIGlzIG5vdCBzZXQiCglleGl0IDEKZmkKCmZ1bmN0aW9uIGNhbGwoKSB7CglQQVlMT0FEPSIkMSIKCVtbICRDQUxMQkFDS19VUkwgPX4gXiguKikvc3RhdHVzKC8pPyQgXV0gfHwgQ0FMTEJBQ0tfVVJMPSIke0NBTExCQUNLX1VSTH0vc3RhdHVzIgoJY3VybCAtLXJldHJ5IDUgLS1yZXRyeS1kZWxheSA1IC0tcmV0cnktY29ubnJlZnVzZWQgLS1mYWlsIC1zIC1YIFBPU1QgLWQgIiR7UEFZTE9BRH0iIC1IICdBY2NlcHQ6IGFwcGxpY2F0aW9uL2pzb24nIC1IICJBdXRob3JpemF0aW9uOiBCZWFyZXIgJHtCRUFSRVJfVE9LRU59IiAiJHtDQUxMQkFDS19VUkx9IiB8fCBlY2hvICJmYWlsZWQgdG8gY2FsbCBob21lOiBleGl0IGNvZGUgKCQ/KSIKfQoKZnVuY3Rpb24gc3lzdGVtSW5mbygpIHsKCWlmIFsgLWYgIi9ldGMvb3MtcmVsZWFzZSIgXTt0aGVuCgkJLiAvZXRjL29zLXJlbGVhc2UKCWZpCglPU19OQU1FPSR7TkFNRTotIiJ9CglPU19WRVJTSU9OPSR7VkVSU0lPTl9JRDotIiJ9CglBR0VOVF9JRD0kezE6LW51bGx9CgkjIHN0cmlwIHN0YXR1cyBmcm9tIHRoZSBjYWxsYmFjayB1cmwKCVtbICRDQUxMQkFDS19VUkwgPX4gXiguKikvc3RhdHVzKC8pPyQgXV0gJiYgQ0FMTEJBQ0tfVVJMPSIke0JBU0hfUkVNQVRDSFsxXX0iIHx8IHRydWUKCVNZU0lORk9fVVJMPSIke0NBTExCQUNLX1VSTH0vc3lzdGVtLWluZm8vIgoJUEFZTE9BRD0ie1wib3NfbmFtZVwiOiBcIiRPU19OQU1FXCIsIFwib3NfdmVyc2lvblwiOiBcIiRPU19WRVJTSU9OXCIsIFwiYWdlbnRfaWRcIjogJEFHRU5UX0lEfSIKCWN1cmwgLS1yZXRyeSA1IC0tcmV0cnktZGVsYXkgNSAtLXJldHJ5LWNvbm5yZWZ1c2VkIC0tZmFpbCAtcyAtWCBQT1NUIC1kICIke1BBWUxPQUR9IiAtSCAnQWNjZXB0OiBhcHBsaWNhdGlvbi9qc29uJyAtSCAiQXV0aG9yaXphdGlvbjogQmVhcmVyICR7QkVBUkVSX1RPS0VOfSIgIiR7U1lTSU5GT19VUkx9IiB8fCB0cnVlCn0KCmZ1bmN0aW9uIHNlbmRTdGF0dXMoKSB7CglNU0c9IiQxIgoJY2FsbCAie1wic3RhdHVzXCI6IFwiaW5zdGFsbGluZ1wiLCBcIm1lc3NhZ2VcIjogXCIkTVNHXCJ9Igp9CgpmdW5jdGlvbiBzdWNjZXNzKCkgewoJTVNHPSIkMSIKCUlEPSR7MjotbnVsbH0KCWNhbGwgIntcInN0YXR1c1wiOiBcImlkbGVcIiwgXCJtZXNzYWdlXCI6IFwiJE1TR1wiLCBcImFnZW50X2lkXCI6ICRJRH0iCn0KCmZ1bmN0aW9uIGZhaWwoKSB7CglNU0c9IiQxIgoJY2FsbCAie1wic3RhdHVzXCI6IFwiZmFpbGVkXCIsIFwibWVzc2FnZVwiOiBcIiRNU0dcIn0iCglleGl0IDEKfQoKIyBUaGlzIHdpbGwgZWNobyB0aGUgdmVyc2lvbiBudW1iZXIgaW4gdGhlIGZpbGVuYW1lLiBHaXZlbiBhIGZpbGUgbmFtZSBsaWtlOiBhY3Rpb25zLXJ1bm5lci1vc3gteDY0LTIuMjk5LjEudGFyLmd6CiMgdGhpcyB3aWxsIG91dHB1dDogMi4yOTkuMQpmdW5jdGlvbiBnZXRSdW5uZXJWZXJzaW9uKCkgewoJRklMRU5BTUU9Int7IC5GaWxlTmFtZSB9fSIKCVtbICRGSUxFTkFNRSA9fiAoWzAtOV0rXC5bMC05XStcLlswLTkrXSkgXV0KCWVjaG8gJEJBU0hfUkVNQVRDSAp9CgpmdW5jdGlvbiBnZXRDYWNoZWRUb29sc1BhdGgoKSB7CglDQUNIRURfUlVOTkVSPSIvb3B0L2NhY2hlL2FjdGlvbnMtcnVubmVyL2xhdGVzdCIKCWlmIFsgLWQgIiRDQUNIRURfUlVOTkVSIiBdO3RoZW4KCQllY2hvICIkQ0FDSEVEX1JVTk5FUiIKCQlyZXR1cm4gMAoJZmkKCglWRVJTSU9OPSQoZ2V0UnVubmVyVmVyc2lvbikKCWlmIFsgLXogIiRWRVJTSU9OIiBdOyB0aGVuCgkJcmV0dXJuIDAKCWZpCgoJQ0FDSEVEX1JVTk5FUj0iL29wdC9jYWNoZS9hY3Rpb25zLXJ1bm5lci8kVkVSU0lPTiIKCWlmIFsgLWQgIiRDQUNIRURfUlVOTkVSIiBdO3RoZW4KCQllY2hvICIkQ0FDSEVEX1JVTk5FUiIKCQlyZXR1cm4gMAoJZmkKCXJldHVybiAwCn0KCmZ1bmN0aW9uIGRvd25sb2FkQW5kRXh0cmFjdFJ1bm5lcigpIHsKCXNlbmRTdGF0dXMgImRvd25sb2FkaW5nIHRvb2xzIGZyb20ge3sgLkRvd25sb2FkVVJMIH19IgoJaWYgWyAhIC16ICJ7eyAuVGVtcERvd25sb2FkVG9rZW4gfX0iIF07IHRoZW4KCVRFTVBfVE9LRU49IkF1dGhvcml6YXRpb246IEJlYXJlciB7eyAuVGVtcERvd25sb2FkVG9rZW4gfX0iCglmaQoJY3VybCAtLXJldHJ5IDUgLS1yZXRyeS1kZWxheSA1IC0tcmV0cnktY29ubnJlZnVzZWQgLS1mYWlsIC1MIC1IICIke1RFTVBfVE9LRU59IiAtbyAiL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L3t7IC5GaWxlTmFtZSB9fSIgInt7IC5Eb3dubG9hZFVSTCB9fSIgfHwgZmFpbCAiZmFpbGVkIHRvIGRvd25sb2FkIHRvb2xzIgoJbWtkaXIgLXAgL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyIHx8IGZhaWwgImZhaWxlZCB0byBjcmVhdGUgYWN0aW9ucy1ydW5uZXIgZm9sZGVyIgoJc2VuZFN0YXR1cyAiZXh0cmFjdGluZyBydW5uZXIiCgl0YXIgeGYgIi9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS97eyAuRmlsZU5hbWUgfX0iIC1DIC9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS9hY3Rpb25zLXJ1bm5lci8gfHwgZmFpbCAiZmFpbGVkIHRvIGV4dHJhY3QgcnVubmVyIgoJIyBjaG93biB7eyAuUnVubmVyVXNlcm5hbWUgfX06e3sgLlJ1bm5lckdyb3VwIH19IC1SIC9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS9hY3Rpb25zLXJ1bm5lci8gfHwgZmFpbCAiZmFpbGVkIHRvIGNoYW5nZSBvd25lciIKfQoKQ0FDSEVEX1JVTk5FUj0kKGdldENhY2hlZFRvb2xzUGF0aCkKaWYgWyAteiAiJENBQ0hFRF9SVU5ORVIiIF07dGhlbgoJZG93bmxvYWRBbmRFeHRyYWN0UnVubmVyCglzZW5kU3RhdHVzICJpbnN0YWxsaW5nIGRlcGVuZGVuY2llcyIKCWNkIC9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS9hY3Rpb25zLXJ1bm5lcgoJc3VkbyAuL2Jpbi9pbnN0YWxsZGVwZW5kZW5jaWVzLnNoIHx8IGZhaWwgImZhaWxlZCB0byBpbnN0YWxsIGRlcGVuZGVuY2llcyIKZWxzZQoJc2VuZFN0YXR1cyAidXNpbmcgY2FjaGVkIHJ1bm5lciBmb3VuZCBpbiAkQ0FDSEVEX1JVTk5FUiIKCXN1ZG8gY3AgLWEgIiRDQUNIRURfUlVOTkVSIiAgIi9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS9hY3Rpb25zLXJ1bm5lciIKCXN1ZG8gY2hvd24ge3sgLlJ1bm5lclVzZXJuYW1lIH19Ont7IC5SdW5uZXJHcm91cCB9fSAtUiAiL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyIiB8fCBmYWlsICJmYWlsZWQgdG8gY2hhbmdlIG93bmVyIgoJY2QgL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyCmZpCgoKc2VuZFN0YXR1cyAiY29uZmlndXJpbmcgcnVubmVyIgp7ey0gaWYgLlVzZUpJVENvbmZpZyB9fQpmdW5jdGlvbiBnZXRSdW5uZXJGaWxlKCkgewoJY3VybCAtLXJldHJ5IDUgLS1yZXRyeS1kZWxheSA1IFwKCQktLXJldHJ5LWNvbm5yZWZ1c2VkIC0tZmFpbCAtcyBcCgkJLVggR0VUIC1IICdBY2NlcHQ6IGFwcGxpY2F0aW9uL2pzb24nIFwKCQktSCAiQXV0aG9yaXphdGlvbjogQmVhcmVyICR7QkVBUkVSX1RPS0VOfSIgXAoJCSIke01FVEFEQVRBX1VSTH0vJDEiIC1vICIkMiIKfQoKc2VuZFN0YXR1cyAiZG93bmxvYWRpbmcgSklUIGNyZWRlbnRpYWxzIgpnZXRSdW5uZXJGaWxlICJjcmVkZW50aWFscy9ydW5uZXIiICIvaG9tZS97eyAuUnVubmVyVXNlcm5hbWUgfX0vYWN0aW9ucy1ydW5uZXIvLnJ1bm5lciIgfHwgZmFpbCAiZmFpbGVkIHRvIGdldCBydW5uZXIgZmlsZSIKZ2V0UnVubmVyRmlsZSAiY3JlZGVudGlhbHMvY3JlZGVudGlhbHMiICIvaG9tZS97eyAuUnVubmVyVXNlcm5hbWUgfX0vYWN0aW9ucy1ydW5uZXIvLmNyZWRlbnRpYWxzIiB8fCBmYWlsICJmYWlsZWQgdG8gZ2V0IGNyZWRlbnRpYWxzIGZpbGUiCmdldFJ1bm5lckZpbGUgImNyZWRlbnRpYWxzL2NyZWRlbnRpYWxzX3JzYXBhcmFtcyIgIi9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS9hY3Rpb25zLXJ1bm5lci8uY3JlZGVudGlhbHNfcnNhcGFyYW1zIiB8fCBmYWlsICJmYWlsZWQgdG8gZ2V0IGNyZWRlbnRpYWxzX3JzYXBhcmFtcyBmaWxlIgpnZXRSdW5uZXJGaWxlICJzeXN0ZW0vc2VydmljZS1uYW1lIiAiL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyLy5zZXJ2aWNlIiB8fCBmYWlsICJmYWlsZWQgdG8gZ2V0IHNlcnZpY2UgbmFtZSBmaWxlIgpzZWQgLWkgJ3MvJC9cLnNlcnZpY2UvJyAvaG9tZS97eyAuUnVubmVyVXNlcm5hbWUgfX0vYWN0aW9ucy1ydW5uZXIvLnNlcnZpY2UKClNWQ19OQU1FPSQoY2F0IC9ob21lL3t7IC5SdW5uZXJVc2VybmFtZSB9fS9hY3Rpb25zLXJ1bm5lci8uc2VydmljZSkKCnNlbmRTdGF0dXMgImdlbmVyYXRpbmcgc3lzdGVtZCB1bml0IGZpbGUiCmdldFJ1bm5lckZpbGUgInN5c3RlbWQvdW5pdC1maWxlP3J1bkFzVXNlcj17eyAuUnVubmVyVXNlcm5hbWUgfX0iICIkU1ZDX05BTUUiIHx8IGZhaWwgImZhaWxlZCB0byBnZXQgc2VydmljZSBmaWxlIgpzdWRvIG12ICRTVkNfTkFNRSAvZXRjL3N5c3RlbWQvc3lzdGVtLyB8fCBmYWlsICJmYWlsZWQgdG8gbW92ZSBzZXJ2aWNlIGZpbGUiCnN1ZG8gY2hvd24gcm9vdDpyb290IC9ldGMvc3lzdGVtZC9zeXN0ZW0vJFNWQ19OQU1FIHx8IGZhaWwgImZhaWxlZCB0byBjaGFuZ2Ugb3duZXIiCmlmIFsgLWUgIi9zeXMvZnMvc2VsaW51eCIgXTt0aGVuCglzdWRvIGNoY29uIC1oIHN5c3RlbV91Om9iamVjdF9yOnN5c3RlbWRfdW5pdF9maWxlX3Q6czAgL2V0Yy9zeXN0ZW1kL3N5c3RlbS8kU1ZDX05BTUUgfHwgZmFpbCAiZmFpbGVkIHRvIGNoYW5nZSBzZWxpbnV4IGNvbnRleHQiCmZpCgpzZW5kU3RhdHVzICJlbmFibGluZyBydW5uZXIgc2VydmljZSIKY3AgL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyL2Jpbi9ydW5zdmMuc2ggL2hvbWUve3sgLlJ1bm5lclVzZXJuYW1lIH19L2FjdGlvbnMtcnVubmVyLyB8fCBmYWlsICJmYWlsZWQgdG8gY29weSBydW5zdmMuc2giCnN1ZG8gY2hvd24ge3sgLlJ1bm5lclVzZXJuYW1lIH19Ont7IC5SdW5uZXJHcm91cCB9fSAtUiAvaG9tZS97eyAuUnVubmVyVXNlcm5hbWUgfX0gfHwgZmFpbCAiZmFpbGVkIHRvIGNoYW5nZSBvd25lciIKc3VkbyBzeXN0ZW1jdGwgZGFlbW9uLXJlbG9hZCB8fCBmYWlsICJmYWlsZWQgdG8gcmVsb2FkIHN5c3RlbWQiCnN1ZG8gc3lzdGVtY3RsIGVuYWJsZSAkU1ZDX05BTUUKe3stIGVsc2V9fQoKR0lUSFVCX1RPS0VOPSQoY3VybCAtLXJldHJ5IDUgLS1yZXRyeS1kZWxheSA1IC0tcmV0cnktY29ubnJlZnVzZWQgLS1mYWlsIC1zIC1YIEdFVCAtSCAnQWNjZXB0OiBhcHBsaWNhdGlvbi9qc29uJyAtSCAiQXV0aG9yaXphdGlvbjogQmVhcmVyICR7QkVBUkVSX1RPS0VOfSIgIiR7TUVUQURBVEFfVVJMfS9ydW5uZXItcmVnaXN0cmF0aW9uLXRva2VuLyIpCgpzZXQgK2UKYXR0ZW1wdD0xCndoaWxlIHRydWU7IGRvCglFUlJPVVQ9JChta3RlbXApCgl7ey0gaWYgLkdpdEh1YlJ1bm5lckdyb3VwIH19CgkuL2NvbmZpZy5zaCAtLXVuYXR0ZW5kZWQgLS11cmwgInt7IC5SZXBvVVJMIH19IiAtLXRva2VuICIkR0lUSFVCX1RPS0VOIiAtLXJ1bm5lcmdyb3VwIHt7LkdpdEh1YlJ1bm5lckdyb3VwfX0gLS1uYW1lICJ7eyAuUnVubmVyTmFtZSB9fSIgLS1sYWJlbHMgInt7IC5SdW5uZXJMYWJlbHMgfX0iIC0tbm8tZGVmYXVsdC1sYWJlbHMgLS1lcGhlbWVyYWwgMj4kRVJST1VUCgl7ey0gZWxzZX19CgkuL2NvbmZpZy5zaCAtLXVuYXR0ZW5kZWQgLS11cmwgInt7IC5SZXBvVVJMIH19IiAtLXRva2VuICIkR0lUSFVCX1RPS0VOIiAtLW5hbWUgInt7IC5SdW5uZXJOYW1lIH19IiAtLWxhYmVscyAie3sgLlJ1bm5lckxhYmVscyB9fSIgLS1uby1kZWZhdWx0LWxhYmVscyAtLWVwaGVtZXJhbCAyPiRFUlJPVVQKCXt7LSBlbmR9fQoJaWYgWyAkPyAtZXEgMCBdOyB0aGVuCgkJcm0gJEVSUk9VVCB8fCB0cnVlCgkJc2VuZFN0YXR1cyAicnVubmVyIHN1Y2Nlc3NmdWxseSBjb25maWd1cmVkIGFmdGVyICRhdHRlbXB0IGF0dGVtcHQocykiCgkJYnJlYWsKCWZpCglMQVNUX0VSUj0kKGNhdCAkRVJST1VUKQoJZWNobyAiJExBU1RfRVJSIgoKCSMgaWYgdGhlIHJ1bm5lciBpcyBhbHJlYWR5IGNvbmZpZ3VyZWQsIHJlbW92ZSBpdCBhbmQgdHJ5IGFnYWluLiBJbiB0aGUgcGFzdCBjb25maWd1cmluZyBhIHJ1bm5lcgoJIyBtYW5hZ2VkIHRvIHJlZ2lzdGVyIGl0IGJ1dCB0aW1lZCBvdXQgbGF0ZXIsIHJlc3VsdGluZyBpbiBhbiBlcnJvci4KCS4vY29uZmlnLnNoIHJlbW92ZSAtLXRva2VuICIkR0lUSFVCX1RPS0VOIiB8fCB0cnVlCgoJaWYgWyAkYXR0ZW1wdCAtZ3QgNSBdO3RoZW4KCQlybSAkRVJST1VUIHx8IHRydWUKCQlmYWlsICJmYWlsZWQgdG8gY29uZmlndXJlIHJ1bm5lcjogJExBU1RfRVJSIgoJZmkKCglzZW5kU3RhdHVzICJmYWlsZWQgdG8gY29uZmlndXJlIHJ1bm5lciAoYXR0ZW1wdCAkYXR0ZW1wdCk6ICRMQVNUX0VSUiAocmV0cnlpbmcgaW4gNSBzZWNvbmRzKSIKCWF0dGVtcHQ9JCgoYXR0ZW1wdCsxKSkKCXJtICRFUlJPVVQgfHwgdHJ1ZQoJc2xlZXAgNQpkb25lCnNldCAtZQoKc2VuZFN0YXR1cyAiaW5zdGFsbGluZyBydW5uZXIgc2VydmljZSIKc3VkbyAuL3N2Yy5zaCBpbnN0YWxsIHt7IC5SdW5uZXJVc2VybmFtZSB9fSB8fCBmYWlsICJmYWlsZWQgdG8gaW5zdGFsbCBzZXJ2aWNlIgp7ey0gZW5kfX0KCmlmIFsgLWUgIi9zeXMvZnMvc2VsaW51eCIgXTt0aGVuCglzdWRvIGNoY29uIC1SIC1oIHVzZXJfdTpvYmplY3RfcjpiaW5fdDpzMCAvaG9tZS9ydW5uZXIvIHx8IGZhaWwgImZhaWxlZCB0byBjaGFuZ2Ugc2VsaW51eCBjb250ZXh0IgpmaQoKQUdFTlRfSUQ9IiIKe3stIGlmIC5Vc2VKSVRDb25maWcgfX0Kc3VkbyBzeXN0ZW1jdGwgc3RhcnQgJFNWQ19OQU1FIHx8IGZhaWwgImZhaWxlZCB0byBzdGFydCBzZXJ2aWNlIgp7ey0gZWxzZX19CnNlbmRTdGF0dXMgInN0YXJ0aW5nIHNlcnZpY2UiCnN1ZG8gLi9zdmMuc2ggc3RhcnQgfHwgZmFpbCAiZmFpbGVkIHRvIHN0YXJ0IHNlcnZpY2UiCgpzZXQgK2UKQUdFTlRfSUQ9JChncmVwICJhZ2VudElkIiAvaG9tZS97eyAuUnVubmVyVXNlcm5hbWUgfX0vYWN0aW9ucy1ydW5uZXIvLnJ1bm5lciB8ICB0ciAtZCAtYyAwLTkpCmlmIFsgJD8gLW5lIDAgXTt0aGVuCglmYWlsICJmYWlsZWQgdG8gZ2V0IGFnZW50IElEIgpmaQpzZXQgLWUKe3stIGVuZH19CnN5c3RlbUluZm8gJEFHRU5UX0lECnN1Y2Nlc3MgInJ1bm5lciBzdWNjZXNzZnVsbHkgaW5zdGFsbGVkIiAkQUdFTlRfSUQKCg=="
}

This is essentially a copy of the default script, with one simple difference. I added:

echo "I am the override script: {{ .ExtraContext.Person }} - {{ .ExtraContext.Who }}" | sudo tee /garm-override.txt

At the top of the script. This should result in a file being created in /garm-override.txt with contents from extra_context.

Hdom commented 5 months ago

Hey @gabriel-samfira, sorry I haven't had the chance to return to this.

I have updated the gcp provider to the latest version, set up the enable_boot_debug, and added cloud-init to our image.

Still having the same issue, here are the logs:

+ CALLBACK_URL=https://[URL-REMOVED]/api/v1/callbacks
+ METADATA_URL=https://[URL-REMOVED]/api/v1/metadata
+ BEARER_TOKEN=[TOKEN-REMOVED].
+ '[' -z https://[URL-REMOVED]/api/v1/metadata ']'
++ getCachedToolsPath
++ CACHED_RUNNER=/opt/cache/actions-runner/latest
++ '[' -d /opt/cache/actions-runner/latest ']'
++ echo /opt/cache/actions-runner/latest
++ return 0
+ CACHED_RUNNER=/opt/cache/actions-runner/latest
+ '[' -z /opt/cache/actions-runner/latest ']'
+ sendStatus 'using cached runner found in /opt/cache/actions-runner/latest'
+ MSG='using cached runner found in /opt/cache/actions-runner/latest'
+ call '{"status": "installing", "message": "using cached runner found in /opt/cache/actions-runner/latest"}'
+ PAYLOAD='{"status": "installing", "message": "using cached runner found in /opt/cache/actions-runner/latest"}'
+ [[ https://[URL-REMOVED]/api/v1/callbacks =~ ^(.*)/status(/)?$ ]]
+ CALLBACK_URL=https://[URL-REMOVED]/api/v1/callbacks/status
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "using cached runner found in /opt/cache/actions-runner/latest"}' -H 'Accept: application/json' -H 'Authorization: Bearer 
+ sudo cp -a /opt/cache/actions-runner/latest /home/runner/actions-runner
+ sudo chown runner:runner -R /home/runner/actions-runner
+ cd /home/runner/actions-runner
+ sendStatus 'configuring runner'
+ MSG='configuring runner'
+ call '{"status": "installing", "message": "configuring runner"}'
+ PAYLOAD='{"status": "installing", "message": "configuring runner"}'
+ [[ https://[URL-REMOVED]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "configuring runner"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ sendStatus 'downloading JIT credentials'
+ MSG='downloading JIT credentials'
+ call '{"status": "installing", "message": "downloading JIT credentials"}'
+ PAYLOAD='{"status": "installing", "message": "downloading JIT credentials"}'
+ [[ https://[URL-REMOVED]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "downloading JIT credentials"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ getRunnerFile credentials/runner /home/runner/actions-runner/.runner
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ getRunnerFile credentials/credentials /home/runner/actions-runner/.credentials
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ getRunnerFile credentials/credentials_rsaparams /home/runner/actions-runner/.credentials_rsaparams
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ getRunnerFile system/service-name /home/runner/actions-runner/.service
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ sed -i 's/$/\.service/' /home/runner/actions-runner/.service
++ cat /home/runner/actions-runner/.service
+ SVC_NAME=actions.runner.[ORG-REMOVED].[RUNNER-NAME-REMOVED].service
+ sendStatus 'generating systemd unit file'
+ MSG='generating systemd unit file'
+ call '{"status": "installing", "message": "generating systemd unit file"}'
+ PAYLOAD='{"status": "installing", "message": "generating systemd unit file"}'
+ [[ https://[URL-REMOVED]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "generating systemd unit file"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ getRunnerFile 'systemd/unit-file?runAsUser=runner' actions.runner.[ORG-REMOVED].[RUNNER-NAME-REMOVED].service
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ sudo mv actions.runner.[ORG-REMOVED].[RUNNER-NAME-REMOVED].service /etc/systemd/system/
+ sudo chown root:root /etc/systemd/system/actions.runner.[ORG-REMOVED].[RUNNER-NAME-REMOVED].service
+ '[' -e /sys/fs/selinux ']'
+ sendStatus 'enabling runner service'
+ MSG='enabling runner service'
+ call '{"status": "installing", "message": "enabling runner service"}'
+ PAYLOAD='{"status": "installing", "message": "enabling runner service"}'
+ [[ https://[URL-REMOVED]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "enabling runner service"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ cp /home/runner/actions-runner/bin/runsvc.sh /home/runner/actions-runner/
cp: cannot stat '/home/runner/actions-runner/bin/runsvc.sh': No such file or directory
+ fail 'failed to copy runsvc.sh'
+ MSG='failed to copy runsvc.sh'
+ call '{"status": "failed", "message": "failed to copy runsvc.sh"}'
+ PAYLOAD='{"status": "failed", "message": "failed to copy runsvc.sh"}'
+ [[ https://[URL-REMOVED]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "failed", "message": "failed to copy runsvc.sh"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN-REMOVED].
+ exit 1
ci-info: no authorized SSH keys fingerprints found for user runner.
Cloud-init v. 20.4.1 finished at Thu, 27 Jun 2024 15:03:12 +0000. Datasource DataSourceGCE.  Up 49.32 seconds
Hdom commented 5 months ago

As mentioned in the OP https://github.com/cloudbase/garm/issues/257#issue-2338907319

I added /. to the cache copy statement inside of the runner_install_template and was able to get a successful runner using the cache.

Although this is a sufficient workaround, we have a lot of pools and id rather not have this code hardcoded into the extra specs of every pool, is there any way to confirm that adding /. to the cache copy wont cause any issues for your image and just implement it into the common template.

sudo cp -a "$CACHED_RUNNER/."  "/home/{{ .RunnerUsername }}/actions-runner"
gc runner list 1f13ebe4-95f3-4ab4-8eaf-0af0182fa60a
+----+------------------------+---------+---------------+--------------------------------------+
| NR | NAME                   | STATUS  | RUNNER STATUS | POOL ID                              |
+----+------------------------+---------+---------------+--------------------------------------+
|  1 | gtd4vlmmn-Cqz4nWhEsp6f | running | idle          | 1f13ebe4-95f3-4ab4-8eaf-0af0182fa60a |
+----+------------------------+---------+---------------+--------------------------------------+
+ CALLBACK_URL=https://[URL]/api/v1/callbacks
+ METADATA_URL=https://[URL]/api/v1/metadata
+ BEARER_TOKEN=[TOKEN]
+ '[' -z https://[URL]/api/v1/metadata ']'
++ getCachedToolsPath
++ CACHED_RUNNER=/opt/cache/actions-runner/latest
++ '[' -d /opt/cache/actions-runner/latest ']'
++ echo /opt/cache/actions-runner/latest
++ return 0
+ CACHED_RUNNER=/opt/cache/actions-runner/latest
+ '[' -z /opt/cache/actions-runner/latest ']'
+ sendStatus 'using cached runner found in /opt/cache/actions-runner/latest'
+ MSG='using cached runner found in /opt/cache/actions-runner/latest'
+ call '{"status": "installing", "message": "using cached runner found in /opt/cache/actions-runner/latest"}'
+ PAYLOAD='{"status": "installing", "message": "using cached runner found in /opt/cache/actions-runner/latest"}'
+ [[ https://[URL]/api/v1/callbacks =~ ^(.*)/status(/)?$ ]]
+ CALLBACK_URL=https://[URL]/api/v1/callbacks/status
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "using cached runner found in /opt/cache/actions-runner/latest"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/status
+ sudo cp -a /opt/cache/actions-runner/latest/. /home/runner/actions-runner
+ sudo chown runner:runner -R /home/runner/actions-runner
+ cd /home/runner/actions-runner
+ sendStatus 'configuring runner'
+ MSG='configuring runner'
+ call '{"status": "installing", "message": "configuring runner"}'
+ PAYLOAD='{"status": "installing", "message": "configuring runner"}'
+ [[ https://[URL]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "configuring runner"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/status
+ sendStatus 'downloading JIT credentials'
+ MSG='downloading JIT credentials'
+ call '{"status": "installing", "message": "downloading JIT credentials"}'
+ PAYLOAD='{"status": "installing", "message": "downloading JIT credentials"}'
+ [[ https://[URL]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "downloading JIT credentials"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/status
+ getRunnerFile credentials/runner /home/runner/actions-runner/.runner
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/metadata/credentials/runner -o /home/runner/actions-runner/.runner
+ getRunnerFile credentials/credentials /home/runner/actions-runner/.credentials
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/metadata/credentials/credentials -o /home/runner/actions-runner/.credentials
+ getRunnerFile credentials/credentials_rsaparams /home/runner/actions-runner/.credentials_rsaparams
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/metadata/credentials/credentials_rsaparams -o /home/runner/actions-runner/.credentials_rsaparams
+ getRunnerFile system/service-name /home/runner/actions-runner/.service
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/metadata/system/service-name -o /home/runner/actions-runner/.service
+ sed -i 's/$/\.service/' /home/runner/actions-runner/.service
++ cat /home/runner/actions-runner/.service
+ SVC_NAME=actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service
+ sendStatus 'generating systemd unit file'
+ MSG='generating systemd unit file'
+ call '{"status": "installing", "message": "generating systemd unit file"}'
+ PAYLOAD='{"status": "installing", "message": "generating systemd unit file"}'
+ [[ https://[URL]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "generating systemd unit file"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/status
+ getRunnerFile 'systemd/unit-file?runAsUser=runner' actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X GET -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' 'https://[URL]/api/v1/metadata/systemd/unit-file?runAsUser=runner' -o actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service
+ sudo mv actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service /etc/systemd/system/
+ sudo chown root:root /etc/systemd/system/actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service
+ '[' -e /sys/fs/selinux ']'
+ sendStatus 'enabling runner service'
+ MSG='enabling runner service'
+ call '{"status": "installing", "message": "enabling runner service"}'
+ PAYLOAD='{"status": "installing", "message": "enabling runner service"}'
+ [[ https://[URL]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "installing", "message": "enabling runner service"}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/status
+ cp /home/runner/actions-runner/bin/runsvc.sh /home/runner/actions-runner/
+ sudo chown runner:runner -R /home/runner
+ sudo systemctl daemon-reload
+ sudo systemctl enable actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service
Created symlink /etc/systemd/system/multi-user.target.wants/actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service → /etc/systemd/system/actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service.
+ '[' -e /sys/fs/selinux ']'
+ AGENT_ID=
+ sudo systemctl start actions.runner.[ORG].gtd4vlmmn-Cqz4nWhEsp6f.service
+ systemInfo
+ '[' -f /etc/os-release ']'
+ . /etc/os-release
++ PRETTY_NAME='Debian GNU/Linux 11 (bullseye)'
++ NAME='Debian GNU/Linux'
++ VERSION_ID=11
++ VERSION='11 (bullseye)'
++ VERSION_CODENAME=bullseye
++ ID=debian
++ HOME_URL=https://www.debian.org/
++ SUPPORT_URL=https://www.debian.org/support
++ BUG_REPORT_URL=https://bugs.debian.org/
+ OS_NAME='Debian GNU/Linux'
+ OS_VERSION=11
+ AGENT_ID=null
+ [[ https://[URL]/api/v1/callbacks/status =~ ^(.*)/status(/)?$ ]]
+ CALLBACK_URL=https://[URL]/api/v1/callbacks
+ SYSINFO_URL=https://[URL]/api/v1/callbacks/system-info/
+ PAYLOAD='{"os_name": "Debian GNU/Linux", "os_version": "11", "agent_id": null}'
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"os_name": "Debian GNU/Linux", "os_version": "11", "agent_id": null}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/system-info/
+ success 'runner successfully installed' null
+ MSG='runner successfully installed'
+ ID=null
+ call '{"status": "idle", "message": "runner successfully installed", "agent_id": null}'
+ PAYLOAD='{"status": "idle", "message": "runner successfully installed", "agent_id": null}'
+ [[ https://[URL]/api/v1/callbacks =~ ^(.*)/status(/)?$ ]]
+ CALLBACK_URL=https://[URL]/api/v1/callbacks/status
+ curl --retry 5 --retry-delay 5 --retry-connrefused --fail -s -X POST -d '{"status": "idle", "message": "runner successfully installed", "agent_id": null}' -H 'Accept: application/json' -H 'Authorization: Bearer [TOKEN]' https://[URL]/api/v1/callbacks/status
ci-info: no authorized SSH keys fingerprints found for user runner.
Cloud-init v. 20.4.1 finished at Thu, 27 Jun 2024 15:34:21 +0000. Datasource DataSourceGCE.  Up 60.81 seconds
gabriel-samfira commented 5 months ago

This is fascinating. Do you by any chance pre-create an empty /home/runner/actions-runner in your template? There is no reason otherwise for the cp command to first create /home/runner/actions-runner and then copy /opt/cache/actions-runner/latest into it.

gabriel-samfira commented 5 months ago

Here is an example of what I mean

image

Hdom commented 5 months ago

This is fascinating. Do you by any chance pre-create an empty /home/runner/actions-runner in your template? There is no reason otherwise for the cp command to first create /home/runner/actions-runner and then copy /opt/cache/actions-runner/latest into it.

@gabriel-samfira Yes! 🤦 that's what it is, we pre-create the folder because we preload some environment variables into the .env file in the actions-runner folder.

gabriel-samfira commented 5 months ago

Closing this for now, as this is not really a bug. We could add a check for the existence of the folder and just copy contents, but in that case we can no longer guarantee that pre-existing contents won't interfere with the functionality of the runner itself (stale libraries, etc).

I suggest any env variable setup be done outside of that folder, or from within the workflow itself.

Feel free to re-open this if needed.

Hdom commented 5 months ago

@gabriel-samfira I resolved the issue by moving the environment file to the cache folder that way the /home/runner/actions-runner folder never gets created and the .env gets copied over from the cache. No issues there.

I suggest any env variable setup be done outside of that folder, or from within the workflow itself.

I haven't been able to set any environment variables (usable by workflow runs) in any of the OS specific environment files, I've tried /etc/environment and such and only the .env file within the runner software folder ensures the environment variables are passed to the workflows. ~There doesn't seem to be any documentation around it, only mention of it I was able to find was this: https://github.com/actions/actions-runner-controller/issues/1135#issuecomment-1046237521~

Actually I found this https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/using-a-proxy-server-with-self-hosted-runners#using-a-env-file-to-set-the-proxy-configuration