Open AurimasNav opened 3 days ago
Hello! Thank you for filing an issue.
The maintainers will triage your issue shortly.
In the meantime, please take a look at the troubleshooting guide for bug reports.
If this is a feature request, please review our contribution guidelines.
@AurimasNav When you tried this without hostNetwork: true
, was it in an environment with a service mesh sidecar injection like istio?
I ran into a similar issue with hostNetwork: true
when 2 dind
runners would come up on the same node at the same time.
One workflow would fail with
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
and the dind
container logs would have
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables: Resource temporarily unavailable.
I think this is because both runners were trying to use iptables
at the same time, for the host network configuration. I suspect using hostNetwork: true
may result in resource contention on the node.
Anyway, I was also using hostNetwork: true
because the containers didn't have internet access without it, which was actually caused by istio sidecar injection. Runners with hostNetwork: true
did not receive istio sidecars, while others did. Any runner with an istio sidecar did not have internet access in containers, and removing the sidecars fixed the "no internet access without hostNetwork
" issue.
@AurimasNav When you tried this without
hostNetwork: true
, was it in an environment with a service mesh sidecar injection like istio?I ran into a similar issue with
hostNetwork: true
when 2dind
runners would come up on the same node at the same time.One workflow would fail with
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
and the
dind
container logs would havefailed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create NAT chain DOCKER: iptables failed: iptables --wait -t nat -N DOCKER: iptables: Resource temporarily unavailable.
I think this is because both runners were trying to use
iptables
at the same time, for the host network configuration. I suspect usinghostNetwork: true
may result in resource contention on the node.Anyway, I was also using
hostNetwork: true
because the containers didn't have internet access without it, which was actually caused by istio sidecar injection. Runners withhostNetwork: true
did not receive istio sidecars, while others did. Any runner with an istio sidecar did not have internet access in containers, and removing the sidecars fixed the "no internet access withouthostNetwork
" issue.
There is no service mesh nor any kind sidecar injection, it is a k3s install on a single node server, but I guess it could potentially be a problem with 2 runners, even though I reduced the max runners to 1 instance, I have another actions runner controller set instance for different github org, running on the same k3s.
Checks
Controller Version
0.9.3
Helm Chart Version
0.9.3
CertManager Version
1.16.1
Deployment Method
ArgoCD
cert-manager installation
cert-manager is working
Checks
Resource Definitions
To Reproduce
Describe the bug
Running an action including docker command like:
docker build . --file Dockerfile --tag $env:FullImageName --secret id=npm_token,env=NPM_TOKEN --build-arg NODE_ENV=production
intermitently results in an error:Describe the expected behavior
Being able to connect to unix:///var/run/docker.sock 100% of the runs.
Whole Controller Logs
Whole Runner Pod Logs
Additional Context
In the dind container log I can see:
Not sure why that happens or how it can be solved? Might this have something to do with my config in values.yaml
(if I don't specify this, my containers in actions have no internet access).