Support sts and static credentials to work together

rpudlowski93 commented 11 months ago

Expected behavior

I would like to set MC_HOST_s3 env using IAM role. I would like to generate temporary credentials for the AWS service.

Actual behavior

I can not install aws cli on minio in order to retrieve temporary credentials so I need to use AWS ACCESS AND SECRET KEY. I would like to use token.

Right now, there is just a feature to retrieve temporary credentials only for minio, but i want s3 bucket.

Steps to reproduce the behavior

I have minio with mc and i want to use my IAM role assigned to Pod in EKS in order to generate temporary credentials for s3 service

mc --version

RELEASE-2023-11-06T04-19-23Z

System information

robert-pudlowski-mox commented 11 months ago

@harshavardhana do You think that it is possible to implement this feature soon?

harshavardhana commented 11 months ago

Yeah it's minor stuff.

robert-pudlowski-mox commented 11 months ago

cool! That's awesome. So in Your opinion, You will be able to implement it and if yes, what is the approximately date of the feature? It is very urgent for our daily basis.

Just for confirmation, I would like to setup MC_HOST_s3 based on IAM roles. Usually I should be able to retrieve the temporary ACCESS/SECRET/TOKEN values using AWS CLI but it is not available on minio image or mc.

harshavardhana commented 11 months ago

fixed in #4763

rpudlowski93 commented 11 months ago

@harshavardhana can You explain me how we can setup the MC_HOST_s3 now? Ho to retrieve the temp creds?

harshavardhana commented 11 months ago

@harshavardhana can You explain me how we can setup the MC_HOST_s3 now? Ho to retrieve the temp creds?

MC_HOST_s3=https://s3.amazonaws.com
MC_WEB_IDENTITY_TOKEN_FILE=
MC_ROLE_ARN=

Obtain web identity token file and role arn

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-oidc

Just like how you would set

AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

You should set the above values for mc.

robert-pudlowski-mox commented 11 months ago

@harshavardhana I added MC_HOST_s3=https://s3.amazonaws.com and I have the envs AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

as well.

Looks it it trying to use them but I'm getting mc: <ERROR> Failed to copyhttp://opencti-minio:9000/opencti-bucket/import/Artifact/xxxxxxx. Insufficient permissions to access this pathhttps://s3.amazonaws.com/xxxxx/opencti-bucket/import/Artifact/xxxxxxxx``

Do You have any idea if something is still missing or maybe the role is not taken properly? I can see in AWS console that the role is not used in my last hours.

robert-pudlowski-mox commented 11 months ago

There is 403 error and Access Denied in debug mode to S3 bucket.

robert-pudlowski-mox commented 11 months ago

Looks like the MC_HOST_s3 is not taking the temporary Access Key, Secret Key and Session Token into account

We have MC_HOST_s3=https://s3.amazonaws.com/ but should be https:/$ACCESS_KEY:$SECRET_KEY:$TOKEN_SESSION@s3.ap-east-1.amazonaws.com

But how to retrieve these values?

harshavardhana commented 11 months ago

@harshavardhana I added MC_HOST_s3=https://s3.amazonaws.com and I have the envs AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxxx:role/s3-access

@robert-pudlowski-mox with mc you must MC_ envs not AWS_ - I had mentioned it clearly that that our values are equivalent of what AWS CLI provides.

MC_HOST_s3=https://s3.amazonaws.com
MC_WEB_IDENTITY_TOKEN_FILE=
MC_ROLE_ARN=

robert-pudlowski-mox commented 11 months ago

@harshavardhana sure! I already checked it as well but nothing has changed.

My minio deployment definition:

ENVS:

name: MC_HOST_s3 value: https://s3.ap-east-1.amazonaws.com
name: MC_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
- name: MC_ROLE_ARN value: arn:aws:iam::xxx:role/xxx-role

But the issue is still the same

robert-pudlowski-mox commented 11 months ago

I have the IAM role assigned to service account which is used by pod. So it should be fine. It works for ILM with the role and service account.

harshavardhana commented 11 months ago

@harshavardhana sure! I already checked it as well but nothing has changed.

My minio deployment definition:

ENVS:

name: MC_HOST_s3 value: https://s3.ap-east-1.amazonaws.com

name: MC_WEB_IDENTITY_TOKEN_FILE value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

name: MC_ROLE_ARN value: arn:aws:iam::xxx:role/xxx-role

But the issue is still the same

When you are doing this can you mc ls --debug and share the output?

robert-pudlowski-mox commented 11 months ago

@harshavardhana

`mc ls --debug s3/xxxx-bucket mc: HEAD / HTTP/1.1 Host: xxxx-bucket.s3.dualstack.ap-east-1.amazonaws.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mc/RELEASE.2023-11-15T22-45-58Z

mc: HTTP/1.1 403 Forbidden Connection: close Content-Type: application/xml Date: Fri, 17 Nov 2023 13:07:27 GMT Server: AmazonS3 X-Amz-Bucket-Region: ap-east-1 X-Amz-Id-2: P/VItbk7pmEJeRpQXkysYdd6enKcApaptywzGAKWJLn2mm3zFrFQg652CNCIvo0047t+Vp9lqls= X-Amz-Request-Id: ZKBVWFC696YS785Y

mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Amazon mc: >> Expires: 2024-03-03 23:59:59 +0000 UTC mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Amazon mc: >> Expires: 2030-08-23 22:21:28 +0000 UTC mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Starfield Technologies, Inc. mc: >> Expires: 2037-12-31 01:00:00 +0000 UTC mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Starfield Technologies, Inc. mc: >> Expires: 2034-06-28 17:39:16 +0000 UTC mc: Response Time: 29.588739ms

mc: HEAD / HTTP/1.1 Host: xxxx-bucket.s3.dualstack.ap-east-1.amazonaws.com User-Agent: MinIO (linux; amd64) minio-go/v7.0.63 mc/RELEASE.2023-11-15T22-45-58Z

mc: HTTP/1.1 403 Forbidden Connection: close Content-Type: application/xml Date: Fri, 17 Nov 2023 13:07:27 GMT Server: AmazonS3 X-Amz-Bucket-Region: ap-east-1 X-Amz-Id-2: gtwD1VlExVFnhd9pxeHKeWXhSf6EI7Qa46lr4xLPoJJN/VbKIe1L1VJkQDZr5OAyY9IWmxcY62U= X-Amz-Request-Id: ZKBXK9CRAGV9R0NG

mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Amazon mc: >> Expires: 2024-03-03 23:59:59 +0000 UTC mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Amazon mc: >> Expires: 2030-08-23 22:21:28 +0000 UTC mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Starfield Technologies, Inc. mc: >> Expires: 2037-12-31 01:00:00 +0000 UTC mc: TLS Certificate found: mc: >> Country: US mc: >> Organization: Starfield Technologies, Inc. mc: >> Expires: 2034-06-28 17:39:16 +0000 UTC mc: Response Time: 4.413231ms

mc: Unable to list folder. Access Denied. (2) ls.go:239 cmd.doList(..) Tags: [https://s3.ap-east-1.amazonaws.com/xxxx-bucket] (1) client-s3.go:2364 cmd.(S3Client).listInRoutine(..) Tags: [xxxx-bucket] (0) client-s3.go:2330 cmd.(S3Client).bucketStat(..) Release-Tag:RELEASE.2023-11-15T22-45-58Z | Commit:4724c024c6de | Host:opencti-minio-8bd97bd7c-5lk65 | OS:linux | Arch:amd64 | Lang:go1.21.4 | Mem:3.0 MiB/15 MiB | Heap:3.0 MiB/7.4 MiB`

rpudlowski93 commented 11 months ago

@harshavardhana ??

harshavardhana commented 11 months ago

And what are the environment variables set before using mc ?

robert-pudlowski-mox commented 11 months ago

@harshavardhana what do You mean?

All the envs which I adding to the deployment:

    env:
    - name: MINIO_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: MINIO_ACCESS_KEY
          name: opencti
    - name: MINIO_SECRET_KEY
      valueFrom:
        secretKeyRef:
          key: MINIO_SECRET_KEY
          name: opencti
    - name: S3_BUCKET_NAME
      valueFrom:
        secretKeyRef:
          key: S3_BUCKET_NAME
          name: opencti
    - name: MC_HOST_s3
      value: https://s3.ap-east-1.amazonaws.com
    - name: MC_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    - name: MC_ROLE_ARN
      value: arn:aws:iam::xx:role/xxxx-role

Nothing more :) There are of course some default environments which comes from EKS. Any idea how to deal with the STS for S3?

harshavardhana commented 11 months ago

And what is the version of mc - can you provide mc --version output?

robert-pudlowski-mox commented 11 months ago

I use the minio image: minio/minio:RELEASE.2023-11-15T20-43-25Z

harshavardhana commented 11 months ago

I use the minio image: minio/minio:RELEASE.2023-11-15T20-43-25Z

Have you used latest mc? https://github.com/minio/mc/releases/tag/RELEASE.2023-11-15T22-45-58Z

The mc inside MinIO container is not the latest

harshavardhana commented 11 months ago

ah I see your problem you have not set the MC_STS_ENDPOINT @robert-pudlowski-mox

harshavardhana commented 11 months ago

In your ENV this must be set to perhaps https://sts.ap-east-1.amazonaws.com

robert-pudlowski-mox commented 11 months ago

@harshavardhana thanks man! it works. One more question. After adding the env MC_STS_ENDPOINT=https://sts.ap-east-1.amazonaws.com, now I can not add alias for local minio deployment which was working before:

mc alias set myminio http://opencti-minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY mc: Configuration written to /tmp/.mc/config.json. Please update your access credentials. mc: Successfully created /tmp/.mc/share. mc: Initialized share uploads /tmp/.mc/share/uploads.json file. mc: Initialized share downloads /tmp/.mc/share/downloads.json file. mc: Unable to initialize new alias from the provided credentials. The Access Key Id you provided does not exist in our records.

Any idea how to make the S3 alias and minio local working?

harshavardhana commented 11 months ago

@robert-pudlowski-mox right now that is not possible - I have to think about adding that support.

robert-pudlowski-mox commented 11 months ago

@harshavardhana so how can I copy files from local minio to S3 bucket if I'm not able to connect in the same time to local minio and S3 bucket? :D

Now I can just send something to S3 bucket or I can only connect to local minio but not both.

robert-pudlowski-mox commented 11 months ago

@harshavardhana and one more question, if I have the configuration setup for S3 bucket (MC_HOST_s3, MC_STS_ENDPOINT etc..) does the ilm which is already in place and setup for local minio and ilm with S3 tier will work? Or it will stop working?

harshavardhana commented 11 months ago

@harshavardhana so how can I copy files from local minio to S3 bucket if I'm not able to connect in the same time to local minio and S3 bucket? :D

Now I can just send something to S3 bucket or I can only connect to local minio but not both.

Correct this is something that needs to be fixed.

harshavardhana commented 11 months ago

@harshavardhana and one more question, if I have the configuration setup for S3 bucket (MC_HOST_s3, MC_STS_ENDPOINT etc..) does the ilm which is already in place and setup for local minio and ilm with S3 tier will work? Or it will stop working?

mc changes won't affect ILM which is server side config.

harshavardhana commented 11 months ago

This is fully fixed here https://github.com/minio/mc/pull/4771

minio / mc