search

fluxcd / flux2-kustomize-helm-example

A GitOps workflow example for multi-env deployments with Flux, Kustomize and Helm.
https://fluxcd.io
Apache License 2.0
990 stars 1.05k forks source link

cannot curl podinfo service - http 503 #80

Closed itapai closed 11 months ago

itapai commented 2 years ago

I forked this repo, added a new cluster in clusters/development and bootstraped flux on a local kind cluster.

flux bootstrap github  \
  --owner=itapai \
  --repository=flux2-kustomize-helm-example \
  --branch=master \
  --path=clusters/development \
  --personal
flux check

► checking prerequisites
✔ Kubernetes 1.25.2 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.25.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.29.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.27.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.30.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

k get no -o wide

NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
local-control-plane   Ready    control-plane   37m   v1.25.2   172.24.0.2    <none>        Ubuntu 22.04.1 LTS   5.19.12-arch1-1   containerd://1.6.8

k get svc -n nginx

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
nginx-ingress-controller                   NodePort    10.96.186.228   <none>        80:32642/TCP,443:30051/TCP   32m
nginx-ingress-controller-default-backend   ClusterIP   10.96.218.142   <none>        80/TCP                       32m

When I try to curl it with the url, I get this error:

curl 172.24.0.2:32642 -H "Host: podinfo.development"

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx</center>
</body>
</html>

and this error log in the nginx pod

k logs -n nginx nginx-ingress-controller-598f55d78c-6pcbq

...
E1003 18:57:16.549786       1 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:nginx:nginx-ingress-controller" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
W1003 18:57:47.749632       1 reflector.go:424] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:nginx:nginx-ingress-controller" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
E1003 18:57:47.749654       1 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:nginx:nginx-ingress-controller" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

However, if I port forward and curl, it works.

 k port-forward -n podinfo svc/podinfo 8088:9898

 curl -H "Host: podinfo.development" localhost:8088
{
  "hostname": "podinfo-6cd678c96-tqqkg",
  "version": "6.2.1",
  "revision": "44157ecd84c0d78b17e4d7b43f2a7bb316372d6c",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v6.2.1",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.19.1",
  "num_goroutine": "9",
  "num_cpu": "8"
}

I tried the same thing with a managed kubernetes service (digitalocean) with a LoadBalancer service, and had the same error in the nginx-ingress pod and curl returned Empty reply from server. How can I access the service via url?

itapai commented 2 years ago

Adding this to the nginx-ingress-controller clusterrole fixed this for me.

- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch

Short

kubectl get clusterrole nginx-ingress-controller -o json | \
  jq '.rules += [{"apiGroups":["discovery.k8s.io"],"resources":["endpointslices"],"verbs":["list","watch"]}]' | \
  kubectl apply -f -