minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.22k stars 455 forks source link

Minio-api-ingress does not work with AKS ingress this is related . #2343 #2346

Closed williamsun-hha closed 3 weeks ago

williamsun-hha commented 1 month ago

Kubernetes ingress part is configured as following

ingress:
  api:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations: { }
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioapi.xyz.abc.com
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations: { }
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioconsole.xyz.abc.com
    path: /
    pathType: Prefix

Expected Behavior

aws s3 ls --endpoint-url https://minioapi.xyz.abc.com --profile s3
2024-10-18 19:41:45 nextgen-s3

However

Connection was closed before we received a valid response from endpoint URL: "https://minioapi.xyz.abc.com/"

The S3 API is working through the service port.

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Regression

Your Environment

ramondeklein commented 1 month ago

I do see two issues:

  1. You specify ingressClassName and that seems to be set to a hostname. However, an ingress class name should refer to an ingress class.
  2. It looks like you specify a secret for TLS, so please check if that secret is loaded with a valid TLS certificate.

Most issues should be visible when you describe the ingress resource...

williamsun-hha commented 1 month ago

Thank you very much for your help!

  1. webapprouting.kubernetes.azure.com is correct ingress class for Azure web app routing ingress. All of other applications are working in this ingress, include the bitnami minio helm chart.
  2. secret is working as well.
  3. Describe ingress does not have any event ( k describe ingress myminio -n tenant-ns
    Name:             myminio
    Labels:           app.kubernetes.io/managed-by=Helm
    Namespace:        tenant-ns
    Address:          17.17.9.11
    Ingress Class:    webapprouting.kubernetes.azure.com
    Default backend:  <default>
    TLS:
    minio-api-tls-cert terminates 
    Rules:
    Host                            Path  Backends
    ----                            ----  --------
    minioapi.xyz.abc.com
                                  /   minio:https-minio (10.244.2.198:9000,10.244.3.176:9000,10.244.0.149:9000 + 1 more...)
    Annotations:                      meta.helm.sh/release-name: tenant
                                  meta.helm.sh/release-namespace: tenant-ns
    Events:                           <none>

    ). The console ingress has no event either. Since Service Load balancer is working on both api and console, so the backend services are working fine. Bitnami MinIO Console and API ingress is working perfectly on the same AKS cluster as well all of the other cluster.

ramondeklein commented 1 month ago

The ingress looks fine at first sight. Can you post the output of curl -v https://minioapi.xyz.abc.com? I see that you are using the https-minio endpoint so the NGINX controller connects to MinIO using TLS too. I can't tell which ingress controller you are using, but you may want to check that too...

williamsun-hha commented 1 month ago

Thank you very much for your help,  Ramon! I got Client sent an HTTP request to an HTTPS server Response 

Sent from Yahoo Mail for iPhone

On Tuesday, October 22, 2024, 12:11 PM, Ramon de Klein @.***> wrote:

The ingress looks fine at first sight. Can you post the output of curl -v https://minioapi.xyz.abc.com? I see that you are using the https-minio endpoint so the NGINX controller connects to MinIO using TLS too. I can't tell which ingress controller you are using, but you may want to check that too...

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

ramondeklein commented 1 month ago

You didn't specify the correct arguments to curl. It looks like you used http and https. Also the returned response seems to be incomplete. Please reply via Github instead of e-mail. Please invoke it as curl -v https://<hostname> and post the output.

williamsun-hha commented 1 month ago

Thank you very much, Ramon! Here is the full message:

ramondeklein commented 1 month ago

It looks like your connection to the Ingress server is fine, but the problem seems to be the connection from the ingress controller to the downstream MinIO server. Your MinIO server seems to be configured to use HTTPS (by default it is), but the ingress controller seems to connect using HTTP. That's probably why it returns Client sent an HTTP request to an HTTPS server..

If you are using NGINX, then you need to add the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation to your ingress controller to ensure that NGINX connects using HTTPS (docs). I'm not sure if NGINX controller will trust the Kubernetes CA by default (I think it will), so you may run into certificate issues, because NGINX controller doesn't trust the MinIO certificate. If you are using another ingress controller, then please check the documentation of the ingress controller on how to connect to the downstream server using HTTPS.

You can add this annotation in the tenant's Helm value as .ingress.api.annotations, so your values.yaml should probably read something like this:

ingress:
  api:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioapi.xyz.abc.com
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioconsole.xyz.abc.com
    path: /
    pathType: Prefix

If you can live without HTTPS within the cluster, then you could also leave it as HTTP and disable HTTPS in MinIO by setting adding the following to the values.yaml:

tenant:
  certificate:
    requestAutoCert: false

This will disable certificates (if you don't specify any other certificates) and should allow NGINX to connect via HTTP.

ramondeklein commented 1 month ago

PS: Please reply via Github instead of via email. Replies via email are not formatted and hard to read.

williamsun-hha commented 4 weeks ago

Thank you very much for your help,  Ramon! I thought the same thing. As soon as I turn off the Minio TLS, console comes right up. I will check the annotations and test TLS. Really appreciate the help and support! You have a great day!

ramondeklein commented 4 weeks ago

@cesnietor @cniackz We may want to fix this in our Helm script. The default setting is that MinIO will generate certificates and it looks like this break ingress. We may want to add the proper annotations if the MinIO server uses HTTPS.

We also may want to raise warnings instead of generating the annotations. The annotations are ingress controller specific, so we could assume NGINX, but some may use Traefik instead.

williamsun-hha commented 4 weeks ago

Thank you very much for your help, Ramon! Really appreciate the support!

I did exactly as below: It works.

tenant:
  certificate:
    requestAutoCert: false

My goal is to have cert-manager to automatically renew the cert on the ingress. I can try to do that from ingress configuration perspective. Do you have a sample config on helm chart values.yaml to share? If not, I will try to get it implemented from ingress perspective.

Thanks again and have a nice day!

williamsun-hha commented 3 weeks ago

Thank you very much for your help,  Ramon!