[Documentation] IRSA example is not working

[kuzm1ch]

/kind bug IRSA example doesn't work as expected. https://kserve.github.io/website/master/modelserving/storage/s3/s3/#create-service-account-with-iam-role

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3access # replace with your IAM role ARN
    serving.kserve.io/s3-endpoint: s3.amazonaws.com # replace with your s3 endpoint e.g minio-service.kubeflow:9000
    serving.kserve.io/s3-usehttps: "1" # by default 1, if testing with minio you can set to 0
    serving.kserve.io/s3-region: "us-east-2"
    serving.kserve.io/s3-useanoncredential: "false" # omitting this is the same as false, if true will ignore provided credential and use anonymous credentials

storage-initializer is not able to pull model when eks.amazonaws.com/role-arn and serving.kserve.io/s3-endpoint annotations used in one SA as in documentation.

  File "/prod_venv/lib/python3.9/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown

For example, this example works fine:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3access # replace with your IAM role ARN

Environment:

• Istio Version: not in scope
• Knative Version: not in scope
• KServe Version: v0.12.0-rc1
• Kubeflow version: not in scope
• Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
• Minikube/Kind version: not in scope
• Kubernetes version: (use kubectl version): not in scope
• OS (e.g. from /etc/os-release):

[dafu-wu]

same issue

[dafu-wu]

@kuzm1ch Remove: serving.kserve.io/s3-endpoint: s3.amazonaws.com serving.kserve.io/s3-usehttps: "1"

then work normally!

[yuzisun]

@kuzm1ch Remove: serving.kserve.io/s3-endpoint: s3.amazonaws.com serving.kserve.io/s3-usehttps: "1"

then work normally!

hmm, these annotations are used to inject the corresponding s3 environment variables for storage initializer.