search

GoogleCloudPlatform / cloudml-samples

Cloud ML Engine repo. Please visit the new Vertex AI samples repo at https://github.com/GoogleCloudPlatform/vertex-ai-samples
https://cloud.google.com/ai-platform/docs/
Apache License 2.0
1.52k stars 859 forks source link

Ability to provide environment variables to container for training #445

Closed jperl closed 4 years ago

jperl commented 5 years ago

Is your feature request related to a problem? Please describe. I would like to be able to provide environment variables when submitting an ai platform training job using a container.

Describe the solution you'd like -e VAR=VALUE -e FOO=BAR ...

andrewferlitsch commented 5 years ago

@jperl

You can set environment variables in your Docker container using ENV and then build the docker container.

We currently do no support dynamically setting (submit) new values for the environment variables when you submit the job via ai-plalform. My recommendation would be to rebuild the docker file with the new environment variable values and then submit the job.

This web link shows an example of using environment variables, building the docker file and then submitting to ai-platform: https://cloud.google.com/ml-engine/docs/custom-containers-training#submit_a_hyperparameter_tuning_job

jperl commented 5 years ago

@andrewferlitsch Thanks for your response.

We need to run many jobs with different environment variables, so building a container per job would take a lot of extra time. I updated our code's configuration to support flags as well, but it would be nice to dynamically pass environment variables instead.

andrewferlitsch commented 4 years ago

@dizcology Could you take a look at this AI platform issue.

jperl commented 4 years ago

We resolved this by making our configuration support flags, so it is no longer an issue. But others may want to support environment variables.

andrewferlitsch commented 4 years ago

Customer (jperl) has a workaround. Will keep open for present time as a placeholder for supporting environment variables.

andrewferlitsch commented 4 years ago

product team aware of feature.

KimRomBot commented 4 years ago

@andrewferlitsch Hello, is there a status/thread on this feature request?

DBCerigo commented 4 years ago

For anyone else looking for work arounds on this, you could use the env command within exec to export variables before running another utility. See man env for details. Ref:https://stackoverflow.com/questions/22869637/setting-variables-with-executing-a-command-with-exec