grailbio / reflow

A language and runtime for distributed, incremental data processing in the cloud
Apache License 2.0
965 stars 52 forks source link

Add note about how to use ~/.aws configurations #35

Closed olgabot closed 6 years ago

olgabot commented 6 years ago

Hello! I'm trying out Go for a project. I ran into some issues with getting Reflow going and found this thread about using AWS credentials, which fixed my problem. This PR adds the text from the issue thread into the README for newcomers.

mariusae commented 6 years ago

Thanks! Pulling these internally; should be pushed to GitHub again soon.

mariusae commented 6 years ago

I think the better fix here is to make this the default. I think most users would expect AWS configuration to be loaded...

mariusae commented 6 years ago

Reading through the code, it does look as if we're already doing this, but only for credentials.

So the fix is to also load region configuration, etc. from the shared configuration.

mariusae commented 6 years ago

Something like this appears to work:

diff --git a/go/src/github.com/grailbio/reflow/config/awsenvconfig/aws.go b/go/src/github.com/grailbio/reflow/config/awsenvconfig/aws.go
index 63963e6cee..520dcabcfe 100644
--- a/go/src/github.com/grailbio/reflow/config/awsenvconfig/aws.go
+++ b/go/src/github.com/grailbio/reflow/config/awsenvconfig/aws.go
@@ -148,8 +148,11 @@ func (c *credentialsSession) AWS() (*session.Session, error) {
            c.err = fmt.Errorf("failed to retrieve AWS credentials: %v", err)
            return
        }
-       c.session, c.err = session.NewSession(&aws.Config{
-           Credentials: credentials.NewCredentials(credProvider),
+       c.session, c.err = session.NewSessionWithOptions(session.Options{
+           Config: aws.Config{
+               Credentials: credentials.NewCredentials(credProvider),
+           },
+           SharedConfigState: session.SharedConfigEnable,
        })
    })
    return c.session, c.err
mariusae commented 6 years ago

(I'll get this through internally; should show up here soon.)

mariusae commented 6 years ago

Fixed in https://github.com/grailbio/reflow/commit/8ec7db690adc615494b8b71f0cb7a42c3c20f9c0