Closed sergeyshaykhullin closed 12 months ago
Hello, I am Blathers. I am here to help you get the issue triaged.
Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.
I have CC'd a few people who may be able to assist you:
If we have not gotten back to your issue within a few business days, you can try the following:
:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.
xref cockroachdb/cockroach#44842
cc @dhartunian this seems to fall between the cracks of server / obs infra. What to do?
@knz any updates on this issue?
We've not found the direct cause of the issue yet, but you might have interest in cockroachdb/cockroach#74329.
We've not found the direct cause of the issue yet, but you might have interest in cockroachdb/cockroach#74329.
Thanks for the information, but it looks that neither of those flags will not resolve the problem with the redirect.
@sergeyshaykhullin can you connect to the web ui from inside the cluster if you do portforwarding and bypass the nginx ingress?
I assume you have something like
external request (https)
-> nginx ingress (tls termination)
-> web ui (http)
and the redirect loop is happening at the CRDB side?
@mihneastaub or @sergeyshaykhullin can you also provide your k8s nginx annotations?
@udnay I'm using istio gateway for the ingress controller.
metadata:
annotations:
kubernetes.io/tls-acme: "true"
meta.helm.sh/release-name: crdb
meta.helm.sh/release-namespace: crdb
nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
nginx.ingress.kubernetes.io/service-upstream: "true"
While starting to take a look into this, I was able to reproduce this issue using the following steps. I figured these steps could be helpful if anyone else is interested in verifying the issue or playing around with different config settings.
Start by creating a few files we'll use for getting everything set up.
You'll need a working K8s cluster. If you don't have one you can run
hack/exec create-cluster
to create a GKE cluster. This assumes you've gotgcloud
installed and configured with an account that can create/manage GKE clusters. The cluster can be deleted withhack/exec delete-cluster
when you're done.
Install cert-manager, ingress-nginx, and the cockroachdb chart by running hack/exec apply
. This can be re-run as needed (idempotent) which can be useful if you change the values or issuer files and want to upgrade the charts.
With everything set up, get a hold of the external IP for the ingress object:
bin/kubectl get ing crdb-cockroachdb-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
cURL the ingress and verify you get an error about maximum redirects reached:
curl -kL --resolve crdb.example.com:443:<LB_IP_FROM_ABOVE> https://crdb.example.com
> curl: (47) Maximum (50) redirects followed
I think this behavior is due to this snippet below which redirects all traffic on the HTTP port to HTTPS except for the healthcheck. @pseudomuto I'm guessing /health
resolves correctly in your example.
Maybe what we should do is inspect X-Forwarded-Proto
and disable redirect when it's set to https
. What do you think @udnay @pseudomuto ? (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto, I assume nginx sets this)
Is there any workaround for this problem?
Any progress here?
Waiting to see a solution for this? A lot of people use ingresses...
Heads up for others; If your ingress is nginx based you can add the nginx.ingress.kubernetes.io/backend-protocol: HTTPS
annotation to the ingress definition and this will fix the issue.
Found this solution here: https://stackoverflow.com/questions/54459015/how-to-configure-ingress-to-direct-traffic-to-an-https-backend-using-https/54459898#54459898
@HcgRandon Works perfectly! Thank you!
NOPE STILL DOESNT WORK
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
meta.helm.sh/release-name: cockroachdb
meta.helm.sh/release-namespace: whocares-prod
traefik.ingress.kubernetes.io/service.serversscheme: https
labels:
app.kubernetes.io/instance: cockroachdb
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: cockroachdb
helm.sh/chart: cockroachdb-7.0.3
name: cockroachdb-ingress
namespace: whocares-prod
spec:
rules:
- host: some.hostname
http:
paths:
- backend:
service:
name: cockroachdb-public
port:
number: 8080
path: /
pathType: Prefix
status:
loadBalancer: {}
@ghost you can try add the ingress configuration in operator deploy file
apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
# this translates to the name of the statefulset that is created
name: cockroachdb
spec:
dataStore:
pvc:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "20Gi"
volumeMode: Filesystem
resources:
requests:
# This is intentionally low to make it work on local kind clusters.
cpu: 2
memory: 4Gi
limits:
cpu: 4
memory: 8Gi
tlsEnabled: true
# You can set either a version of the db or a specific image name
# cockroachDBVersion: v21.2.8
image:
name: cockroachdb/cockroach:v22.1.1
# nodes refers to the number of crdb pods that are created
# via the statefulset
nodes: 3
additionalLabels:
crdb: is-cool
# affinity is a new API field that is behind a feature gate that is
# disabled by default. To enable please see the operator.yaml file.
# The affinity field will accept any podSpec affinity rule.
# affinity:
# podAntiAffinity:
# preferredDuringSchedulingIgnoredDuringExecution:
# - weight: 100
# podAffinityTerm:
# labelSelector:
# matchExpressions:
# - key: app.kubernetes.io/instance
# operator: In
# values:
# - cockroachdb
# topologyKey: kubernetes.io/hostname
# nodeSelectors used to match against
# nodeSelector:
# worker-pool-name: crdb-workers
# Ingress for access the services
ingress:
ui:
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" <-- add here
host: some.hostname
has anyone tried using path based routing instead of host based routing? I am facing issue on /api/v2/sql
and /uiconfig
when using a different path instead of /
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: cockroachdb-ingress
namespace: cockroach
spec:
rules:
- host: shared.host.name
http:
paths:
- backend:
service:
name: croach-cockroachdb-public
port:
number: 8080
path: /cockroach(/|$)(.*)
pathType: ImplementationSpecific
@prafull01 can you triage?
@himanshu-cockroach Can you please have a look
@rajavarapusaiprasanth-hpe The path based routing is not working currently because of this issue: https://github.com/cockroachdb/cockroach/issues/91429. since all the other paths including UI dashboard, other than the absolute paths which are being redirected to via the code, its working through path based routing, we can close this thread.
Describe the problem I am getting
too many redirects
when trying to access secure cockroachdb cluster behind ingress-nginx using helm chart.Please describe the issue you observed, and any steps we can take to reproduce it:
It is happening because tls.enabled == secure cluster with tls termination. Same thing i got with argo cd and solved using --insecure flag (Insecure - just about tls termination) So i have to use insecure cluster behind ingress (losing auth screen, users with passwords etc) or secure cluster without ingress inside k8s (And i have to manage certificates, use not 80 and 443 ports, it's painful)
To Reproduce
Expected behavior I can disable tls termination in web ui, but don't lose security cluster benefits
Additional data / screenshots
Environment:
Jira issue: CRDB-4219