Open ealasgarov opened 1 year ago
P.S. I have applied the same deployment using my user impersonation: kubectl apply -f resource.yaml --as="myserviceprinciple" and it is working, so it cannot be because of permissions, i thinks.
Ok, this is very odd: I have set "private-cluster: false" and it went through, while my cluster is a private cluster, and previously it was working only with "private-cluster: true" parameter. I can see on Azure portal: "Private cluster: Enabled" so nothing changed from that perspective.
Hello @ealasgarov. Thanks for the ping. This is very weird because for private clusters the API server is not publicly accessible. The API server is what kubectl and this action communicate with. The fact that your kubectl apply
command worked seems to imply your cluster wasn't actually a private cluster (which doesn't make sense).
P.S. I have applied the same deployment using my user impersonation: kubectl apply -f resource.yaml --as="myserviceprinciple" and it is working, so it cannot be because of permissions, i thinks.
Was this run on the GitHub actions runner or some other way? Or is your actions runner a self-hosted runner?
Hi @OliverMKing, yes, I'm using self-hosted runners which are running on another aks cluster (github-runner-controller project). It's definitely a private cluster, it's also says so on azure portal as well: "Private cluster Enabled". Something has changed in the behavior recently (after I upgraded from 1.23.8 --> 1.24.9). I think we need someone else with a private cluster of version 1.24.x to test this with k8s-deploy actions (with setting private cluster: false/true), to understand what has changed.
@ealasgarov I will test this in a matching environment when I get the chance.
What I believe is happening is your clusters (both the runner and deploy target) are both in the same vnet. In that case you can access the API server endpoint from the runner. What the private-cluster
toggle does is switch from using just plain kubectl apply...
(and other) commands to using az aks invoke "kubectl apply"
commands. If both clusters are in the same vnet you can just use normal kubectl apply
commands hence why the private-cluster: false
scenario works (and is actually preferred).
This might require a documentation update to make more clear. Hopefully when I reproduce, I find what the actual error with private-cluster: true
is (I'm not sure why az aks invoke
wouldn't work). The error log itself is another bug, we need to add more context to our logs so we don't print meaningless error statements like this.
@ealasgarov I will test this in a matching environment when I get the chance.
What I believe is happening is your clusters (both the runner and deploy target) are both in the same vnet. In that case you can access the API server endpoint from the runner. What the
private-cluster
toggle does is switch from using just plainkubectl apply...
(and other) commands to usingaz aks invoke "kubectl apply"
commands. If both clusters are in the same vnet you can just use normalkubectl apply
commands hence why theprivate-cluster: false
scenario works (and is actually preferred).This might require a documentation update to make more clear. Hopefully when I reproduce, I find what the actual error with
private-cluster: true
is (I'm not sure whyaz aks invoke
wouldn't work). The error log itself is another bug, we need to add more context to our logs so we don't print meaningless error statements like this.
Thank you for your reply, actually they are in different vnets, however there's a peering, so maybe you're right, but then I'm not sure why did it work prior to that... it would be great if you can test it in a similar environment and yes, also more verbose logging would be appreciated. :)
This issue is idle because it has been open for 14 days with no activity.
What happened?
I have upgraded my private cluster to latest stable version 1.24.9, since then cannot get the pipeline to work. (although I'm also using the new service principle [azure credentials], new clusterrole/binding for that service principle, but I guess here there are no issues).
I have deployed in this way previously with no problems, but now getting "error undefined" upon deploy step.
Here's my pipeline:
I am not sure what else could be an issue. I think if the problem was with credentials I would get a different error. @OliverMKing Any ideas perhaps? Many thanks in advance!
Version
Runner
self-hosted on AKS, latest version 2.303
Relevant log output