carl-reverb
commented
6 months ago Well, finding some more time to dig, I went into the source code here and started tracing out the execution path since the stack trace doesn't give many clues as to where this HTTP request failed. Noting that the first thing that probably does a request is: https://github.com/actions/runner-container-hooks/blob/main/packages/k8s/src/k8s/index.ts#L455
I shell into my pod and install node and then attempt this direct basic implementation:
const k8s = require('@kubernetes/client-node');
const kc = new k8s.KubeConfig();
kc.loadFromDefault();
const k8sApi = kc.makeApiClient(k8s.CoreV1Api);
let main = async () => {
try {
const podsRes = await k8sApi.listNamespacedPod('actions-runners');
console.log(podsRes.body);
} catch (err) {
console.error(err);
}
};
main();It fails like so:
{
// ...
body: {
kind: 'Status',
apiVersion: 'v1',
metadata: {},
status: 'Failure',
message: 'Unauthorized',
reason: 'Unauthorized',
code: 401
},
statusCode: 401
}So I can presume that the problem is not the fault of the hooks library, but something is wrong with either the service account or the cluster configuration in EKS. There's not a lot of easily-findable documentation on how to perform in-cluster authentication via service account because most users want to authenticate to their cluster from outside, using eksctl or similar.
The Role is as configured by the helm chart:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: reverbdotcom-general-purpose-gha-rs-kube-mode
namespace: actions-runners
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- create
- delete
- apiGroups:
- ""
resources:
- pods/exec
verbs:
- get
- create
- apiGroups:
- ""
resources:
- pods/log
verbs:
- get
- list
- watch
- apiGroups:
- batch
resources:
- jobs
verbs:
- get
- list
- create
- delete
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- create
- deleteApparently if there is some RBAC issue I should receive a 403. A 401 indicates that the token was rejected completely. I also checked to see if the token in the client configuration matched the one mounted in the pod, and it does.
I'm out of ideas for now... until I can learn more about debugging 401 with an in-cluster service account token.
nikola-jokic
timmjd
sofiegonzalez
tleerai
When I attempt to run a workflow against a self-hosted runner deployed using the
gha-runner-scale-set-controllerandgha-runner-scale-setcharts, my job fails on the 'Initialize Containers' step.Runner Scale Set
values.yaml:In the Github UI after the job is picked up the following error message appears in the log:
Full error context:
My workflow:
I have tried a lot of different things to try to understand what is not working here but the chain of dependencies and effects is not easy to comprehend. There is a lot of red herrings and other noise in the logs which led me on several chases around the web, and I spent a while trying security contexts, various container images, etc. At this point I think I have run out of time to figure this out and will have to fall back to the previous actions runner controller and advise my team that the next generation of actions runners is a risk and we should evaluate alternative CI pipelines.