Example using container on AI Platform with GOOGLE_APPLICATION_CREDENTIALS

[jperl]

Is your feature request related to a problem? Please describe.

We are using a container with AI Platform that needs to write to a private cloud storage bucket. Locally we mount a volume to provide the credentials file as described in: Running locally using Docker with access to GCP services.

How should we provide credentials using the ai platform containers?

Describe the solution you'd like

An example with a container that configures the google application credentials, without copying it in the docker container.

[jperl]

For now we are copying the credentials file into the docker container, which is not ideal for security.

[andrewferlitsch]

You can set the GOOGLE_APPLICATION_CREDENTIALS locally, and the gcloud command will pick up the value from your local terminal session.

[jperl]

Inside the docker container we are using tensorflow's gfile to read/write to gcp file paths (gs://...). So the container needs the credential file.

[andrewferlitsch]

@jperl Hi, it seems this issue got overlooked for awhile. Were you able to resolve the issues with the docker container and credentials file?

[jperl]

Yes by copying the credentials file into the docker container which is not ideal for security, but also not a show stopper.

[andrewferlitsch]

Jon

Thanks for the update.

On Tue, Sep 24, 2019 at 8:52 AM Jon Perl notifications@github.com wrote:

Yes by copying the credentials file into the docker container which is not ideal for security, but also not a show stopper.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/cloudml-samples/issues/442?email_source=notifications&email_token=AFOVC2HAVVNI4EO2MQ2ANALQLIZTRA5CNFSM4H6TMKEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7O3H2I#issuecomment-534623209, or mute the thread https://github.com/notifications/unsubscribe-auth/AFOVC2AGS2AA2F46M74SD5TQLIZTRANCNFSM4H6TMKEA .

[andrewferlitsch]

@jperl I am closing this issue. Features like this are aware by the product team and new features are on product roadmap.

[dynamix]

I just run into the same issue, considering that this has been closed more than six month ago, is there a proper solution for this now? We are submitting the jobs via the REST API and not the glcoud command.

[JulianFerry]

My understanding is that you shouldn't be passing credentials anywhere. Instead you should set up permissions for the cloud-ml-engine service account directly on the IAM admin console.

After activating the AI platform API you should see a service account called service-*@cloud-ml.google.com.iam.gserviceaccount.com. On this page you can add permissions (such as Storage Object Viewer or Storage Object Admin) to your AI platform containers.

[sanealytics]

For local run, this does not work for me.

I have GOOGLE_APPLICATION_CREDENTIALS as suggested by @andrewferlitsch

Locally, I can run a python file and have it access storage, etc fine. But when I try docker run $IMAGE_URI --epochs 1, I get the dreaded

google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

I can pass the credentials as suggested for local testing, but it would be nice to have a solution that makes local testing faster.

Thanks

[cgarciae]

@JulianFerry does this work for custom tier (user defined containers)? According to this e.g. Cloud Run container get the GOOGLE_APPLICATION_CREDENTIALS injected but AI Platform Training is not mentioned.