How to access to S3 bucket in Docker for Sander

[mvarewyck]

Sander's access credentials just require the bucket name on his local PC. He automatically has access to the data in the S3 bucket then.

Possible scenarios on how to connect: https://github.com/cloudyr/aws.signature/

Current script on how to connect to the S3 bucket: https://github.com/inbo/aspbo/blob/main/src/connect_to_bucket.R

Current script for building docker image: https://github.com/inbo/aspbo/blob/14-create-script-to-build-docker-environment/src/build_docker_environment.Rmd

[Yingjie-ZhangOA]

@TheJenne18 We wonder how the connection to aws s3 bucket is done here https://github.com/inbo/aspbo/blob/main/src/connect_to_bucket.R#L51. In order to enable Sander to do local testing, we'll need to adapt this way of setting up connection into our workflow.

In the current implementation from our side, we extract aws_access_key_id, aws_secret_access_key and aws_session_token from ~/.aws/credentials and make all these environmental variables in R.

Does aws-cli-mfa-login.py also generates these attributes for a given user and where it's saved? Thanks.

[mvarewyck]

@SanderDevisscher Given the fact that you have access after running the connect_to_bucket script, I think we're in case 5 or 6 here

Can you check 2 more things?

• On Windows, the default location for the credentials file is C:\Users\. aws\credentials. Is there anything?
• Is there an environment variable AWS_SHARED_CREDENTIALS_FILE?

[SanderDevisscher]

@mvarewyck I have a .aws folder with a credentials file on the specified location. It does contain a aws_session_token.

However there is no AWS_SHARED_CREDENTIALS_FILE environment variable. I can add it to my .renviron file.

[Yingjie-ZhangOA]

@SanderDevisscher

I have a .aws folder with a credentials file on the specified location. It does contain a aws_session_token

Great, that is the file we need. Can you email me the

• the exactly location of C:\Users. aws\credentials
• format of the content of the cendentials file, I expect something like

  - [INBO]
     -  aws_acces_key_id ( or .AccessKeyId)
     - aws_secret_access_key (or .SecretAccessKey)
     - aws_session_token (or .SessionToken)

so that I can parse them and read them into R.

[Yingjie-ZhangOA]

@SanderDevisscher I've updated the connection setup. Can you pull and install the latest version from branch feature#53 and try to start the docker container using the following command

docker run -it -v C:\Users\sander_devisscher\.aws :/root/.aws -p 3838:3838 inbo/alienspecies R -e "alienSpecies::setupS3(inboUser = 'sander_devisscher');alienSpecies::runShiny()"

[SanderDevisscher]

@Yingjie-ZhangOA I had to slightly change to command to work, see below:

docker run -it -v C:/Users/sander_devisscher/.aws:/root/.aws -p 3000:3838 inbo/alienspecies:latest R -e "alienSpecies::setupS3(inboUser = 'sander_devisscher');alienSpecies::runShiny()"

but after starting R I get the following error: https://docs.google.com/document/d/1ut_ZWO2PjF2mD-uTO50f_Eo5vuXBMSUkhQZ5YJKutGs/edit?usp=sharing

[Yingjie-ZhangOA]

Thanks for the feedback, it says you still don't have connection to the bucket, so the credentials are still not read correctly into R. Do you mind checking the output of the following for me?

docker run -it -v C:/Users/sander_devisscher/.aws:/root/.aws -p 3000:3838 inbo/alienspecies:latest R -e "alienSpecies::setupS3(inboUser = 'sander_devisscher');dir('/root/.aws');Sys.getenv(c('AWS_ACCESS_KEY_ID','AWS_SECRET_ACCESS_KEY','AWS_SESSION_TOKEN'))"

Do you see

[1] "config" "credentials"

and if any of the aws credentials is not the same as the ones stored in C:/Users/sander_devisscher/.aws?

[mvarewyck]

@Yingjie-ZhangOA Do you mean sth like this command?

docker run -it -v C:/Users/sander_devisscher/.aws:/root/.aws -p 3000:3838 inbo/alienspecies:latest R -e "alienSpecies::setupS3(inboUser = 'sander_devisscher'); list.files('/root/.aws'); Sys.getenv('AWS_DEFAULT_REGION'); Sys.getenv('AWS_ACCESS_KEY_ID'); Sys.getenv('AWS_SECRET_ACCESS_KEY'); Sys.getenv('AWS_SESSION_TOKEN')"

[Yingjie-ZhangOA]

@Yingjie-ZhangOA Do you mean sth like this command?

docker run -it -v C:/Users/sander_devisscher/.aws:/root/.aws -p 3000:3838 inbo/alienspecies:latest R -e "alienSpecies::setupS3(inboUser = 'sander_devisscher'); list.files('/root/.aws'); Sys.getenv('AWS_DEFAULT_REGION'); Sys.getenv('AWS_ACCESS_KEY_ID'); Sys.getenv('AWS_SECRET_ACCESS_KEY'); Sys.getenv('AWS_SESSION_TOKEN')"

yes, thanks Machteld for correcting, I just realise I didn't paste the correct command.

[SanderDevisscher]

I'll test the command

[SanderDevisscher]

I get my credentials when I run the command however the have "\r" appended on the end.

[mvarewyck]

@SanderDevisscher Could you try again with my latest changes?

If that works, can you then please try

docker run -it -v C:\Users\sander_devisscher\.aws :/root/.aws -p 3838:3838 inbo/alienspecies R -e "alienSpecies::setupS3(inboUser = 'sander_devisscher');alienSpecies::runShiny()"

[SanderDevisscher]

I'm able to run the docker succesfully now 🥳