Ignore host name for signature

[manju-rn]

Expected behavior

I have setup a k8s multicluster setup using linkerd. The way it works is:

1. The service from east cluster will be mirrored to west cluster. To differentiate between services, the service name in the west cluster will be appended with host cluster name.
2. In this case minio service hosted in east cluster will be "mirrored" to west cluster and will be named like minio-east Any pods from within the west cluster can call this service as it is local.
3. The mc client (as pod) is hosted in west cluster and can call minio-east service and connection goes thru fine, but minio server in the east cluster refuses service as signature is not matched. As i understand, minio server uses the original host name for signature which is minio but since mc client is calling as minio-east it refuses. Can mc client override the host entry? OR can minio server ignore the signature verification (since i am anyway calling from mTLS based connection OR is there a way to add additional host names in SERVER_URL env in minio server?

Actual behavior

minio throws signature is not matched due to difference in service name /host name

Steps to reproduce the behavior

1. Install 2 cluster (east and west) and install linkerd with common trust anchor
2. Deploy minio server in east and annotate linkerd tag to create service mirror in west cluster.
3. Deploy MC client as a pod in west cluster and call minio "service mirror" minio-east

mc --version

• (paste output of mc --version) mc version RELEASE.2024-01-18T07-03-39Z (commit-id=bb25267eaadc0e025243b443a94ff33bde6302ad) Runtime: go1.21.6 linux/arm64 Copyright (c) 2015-2024 MinIO, Inc. License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

Microk8s

[harshavardhana]

S3 spec mandates hostnames. We can't change it.

[manju-rn]

S3 spec mandates hostnames. We can't change it.

is there a way to add additional host names in SERVER_URL env in minio server?

[harshavardhana]

is there a way to add additional host names in SERVER_URL env in minio server?

that's a server question not an mc question.

[manju-rn]

is there a way to add additional host names in SERVER_URL env in minio server?

that's a server question not an mc question.

Yes. Correct. Ignore the question here. Will post it there. However, is there a way to override the host name when mc sends the details to mino server? like in curl you can override the host header. Assuming the minio server will use that host header fro signature

[harshavardhana]

is there a way to add additional host names in SERVER_URL env in minio server?

that's a server question not an mc question.

Yes. Correct. Ignore the question here. Will post it there. However, is there a way to override the host name when mc sends the details to mino server? like in curl you can override the host header. Assuming the minio server will use that host header fro signature

No sir, that would be security issue.

[manju-rn]

okay so it looks like minio service than cannot be called from "meshed" cross cluster resource and via ingress is the only option.

[manju-rn]

Okay found the solution. Details here for anybody using linkerd-multicluster Problem Statement: The details mentioned at the start of the post has incorrect statement # 3 . Issue is (re)explained below

3. The mc client (as pod) is hosted in west cluster and can call minio-east service and connection goes thru fine, but minio server in the east cluster refuses service as signature is not matched.
   • Signature creation at client i.e MC client in west cluster: It uses the URL given in the alias setting. In this case, since mirror service (using linkerd-multicluster) is used, the minio URL uses the service name minio-east (mirror of corresponding service name minio located on east cluster)
   • Signature verification at Server i.e Minio server in east cluster: When the minio server receives the request from the MC client, it uses the host name that is received in request header for signature calculation. Due to linkerd-multicluster gateway which intercepts the call from west to east cluster for minio service, it updates the host header from minio-east to minio. Although this is technically correct, the calculation of signature done by MC using minio-east does not match the signature via minio host calculated by minio server.
     Linkerd-multicluster is technically correct since this service mirror concept to connect 2 clusters would need this setup. However, since the host name is changed from when it was generated causes this issue. Now, there is no configuration at linkerd-multicluster to preserve the original host header. However, the solution is Linkerd-SMI using Traffic split. https://linkerd.io/2.14/features/multicluster/# Steps:
   • Define TrafficSplit which will map a dummy service which has exactly same name as was expected in the minio service in east cluster to linkerd mirrored service minio-east
   • Update the alias in the MC client to connect to minio in west cluster - thats it! How does it work:
4. When MC client in west cluster calls the service, it actually uses minio and not minio-east. Traffic spit ensures that it calls the mirror service mini-east and there onto minio service in east cluster. However, since MC client called the minio as host, it will calculate its signature using this host name and the signature will match.

kind: TrafficSplit
apiVersion: split.smi-spec.io/v1alpha2
metadata:
  name: minio-split
  namespace: minio-ns
spec:
  service: minio #Dummy Service with correct name in west cluster just to redirect traffic to mirrored service minio-east
  backends:
  - service: minio-east
    weight: 1000
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio-ns
spec:
  ports:
  - port: 80
    targetPort: 80 #Simple headless mapping to the port of the minio-east service

Hope this helps. The transfer of data via linkerd-SMI between 2 cluster is amazingly fast (compared to connecting via domain hosted URL)! Tested from Cloud to local and vice versa.