Could not find user defined environment variables in the built image

[chengyu-liu-cs]

Hi,

I have been using webservice to deploy trained model and everything works fine. But I had issues when I built a docker image instead of deploying models as webservice. The issue is related to Environment.environment_variables.

I have created environment variables and passed to Environment object of InferenceConfig instance. These variables were used to load models. I was using the same codes to create InferenceConfig instance for webservice and docker image. These variables did exist in the deployed webservice but did not in the built images. I donot understand why these variables could not be found in built images ???

Some of webservice deployment codes ` env_config = RunConfiguration().load(inference_config['environment']).environment # inference_config is typical dict from a yaml file where environment is a yaml file name. add_environment_variables = {'MLMLCM_MODEL_NAMES': model.name, 'MLMLCM_MODEL_VERSIONS': str(model.version} env_config.environment_variables = { env_config.environment_variables, add_environment_variables}

inference_config['environment'] = env_config inference_config = InferenceConfig(**inference_config) inference_config.environment.save_to_directory(path="./saved_env", overwrite=True) service = Model.deploy(workspace=ws, name='AIservice', models=[model], inference_config=inference_config, deployment_config=deployment_config, deployment_target=deployment_target) service.wait_for_creation(show_output=True) `

Some of docker image building codes ` env_config = RunConfiguration().load(inference_config['environment']).environment add_environment_variables = {'MLMLCM_MODEL_NAMES': model.name, 'MLMLCM_MODEL_VERSIONS': str(model.version} env_config.environment_variables = { env_config.environment_variables, add_environment_variables}

inference_config['environment'] = env_config inference_config = InferenceConfig(**inference_config) inference_config.environment.save_to_directory(path="./saved_env", overwrite=True) package = Model.package(workspace=ws, models=[model], inference_config=inference_config, generate_dockerfile=False, image_name='AIservice', image_label=None) package.wait_for_creation(show_output=True) `

I saved InferenConfig.environment to a directory and these environment variables are there in azureml_environment.json file. { "name": "env_scoring", "version": null, "environmentVariables": { "MLMLCM_MODEL_NAMES": "rf-species-cls", "MLMLCM_MODEL_VERSIONS": "25" }, "python": { "userManagedDependencies": false, "interpreterPath": "python", "condaDependenciesFile": "conda_scoring.yml", "baseCondaEnvironment": null }, "docker": { "enabled": true, "baseImage": "mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20200821.v1", "baseDockerfile": null, "sharedVolumes": true, "shmSize": "2g", "arguments": [], "baseImageRegistry": { "address": null, "username": null, "password": null, "registryIdentity": null }, "platform": { "os": "Linux", "architecture": "amd64" } }, "spark": { "repositories": [], "packages": [], "precachePackages": true }, "databricks": { "mavenLibraries": [], "pypiLibraries": [], "rcranLibraries": [], "jarLibraries": [], "eggLibraries": [] }, "r": null, "inferencingStackVersion": null }

output from print(os.environ) in scoring file in the deployed webservice {'KUBERNETES_PORT_443_TCP_PORT': '443', 'KUBERNETES_PORT': 'tcp://10.0.0.1:443', 'WORKSPACE_NAME': 'futechai-mlops-amls-ws', 'WORKER_TIMEOUT': '300', 'KUBERNETES_SERVICE_PORT': '443', 'OLDPWD': '/var/runit/gunicorn', 'AML_MODEL_DC_STORAGE_ENABLED': 'false', 'AZUREML_MODEL_DIR': 'azureml-models', 'KUBERNETES_SERVICE_HOST': '10.0.0.1', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_80_TCP_PORT': '80', 'SCORING_TIMEOUT_MS': '60000', 'LC_ALL': 'C.UTF-8', 'INTEL_MPI_VERSION': '2018.3.222', 'MLMLCM_MODEL_LABELS': 'species_a,species,species_b', 'LD_LIBRARY_PATH': '/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d/lib:/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d/lib:', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_SERVICE_HOST': '10.0.247.208', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_443_TCP_PORT': '443', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_80_TCP': 'tcp://10.0.247.208:80', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_SERVICE_PORT': '443', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_80_TCP_PROTO': 'tcp', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_SERVICE_PORT_443': '443', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_443_TCP_PROTO': 'tcp', 'PATH': '/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/intel/compilers_and_libraries/linux/mpi/bin64', 'MINICONDA_VERSION': '4.7.12.1', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_443_TCP_ADDR': '10.0.247.208', 'AZUREML_ENTRY_SCRIPT': 'score.py', 'PWD': '/var/azureml-app', 'AZUREML_ENVIRONMENT_IMAGE': 'True', 'AZUREML_CONDA_ENVIRONMENT_PATH': '/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d', 'MLMLCM_MODEL_NAMES': 'rf-species-cls', 'LANG': 'C.UTF-8', 'SERVICE_NAME': 'iris-classification', 'COMPUTE_TYPE': 'ACI', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_443_TCP': 'tcp://10.0.247.208:443', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_SERVICE_PORT_80': '80', 'SHLVL': '0', 'HOME': '/root', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT_80_TCP_ADDR': '10.0.247.208', 'PYTHONPATH': '/var/azureml-server:', 'COMPUTE_NAME': 'iris-classification', 'SERVICE_ID': 'd32e1868-5391-4380-875a-65b0142f6da0', 'SVDIR': '/var/runit', 'DEBIAN_FRONTEND': 'noninteractive', 'MLMLCM_MODEL_VERSIONS': '25', 'CAAS_F02C8EA69DE24FBA93566969A55F6BFB_PORT': 'tcp://10.0.247.208:443', 'KUBERNETES_PORT_443_TCP_ADDR': '10.0.0.1', 'COMPUTE_LOCATION': 'northeurope', 'KUBERNETES_PORT_443_TCP': 'tcp://10.0.0.1:443', 'AML_APP_INSIGHTS_ENABLED': 'false', 'SERVER_SOFTWARE': 'gunicorn/19.9.0', 'PYSPARK_SUBMIT_ARGS': 'pyspark-shell', 'KMP_INIT_AT_FORK': 'FALSE'}

output from print(os.environ) in scoring file in the built image {'CAAS_4D736229D6AB4F719927FCFB196F10B4_PORT_80_TCP_PROTO': 'tcp', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'KUBERNETES_PORT': 'tcp://10.0.0.1:443', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_SERVICE_PORT_80': '80', 'WORKER_TIMEOUT': '300', 'KUBERNETES_SERVICE_PORT': '443', 'OLDPWD': '/var/runit/gunicorn', 'AZUREML_MODEL_DIR': 'azureml-models', 'KUBERNETES_SERVICE_HOST': '10.0.0.1', 'LC_ALL': 'C.UTF-8', 'INTEL_MPI_VERSION': '2018.3.222', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_PORT': 'tcp://10.0.202.194:80', 'LD_LIBRARY_PATH': '/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d/lib:/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d/lib:', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_SERVICE_PORT': '80', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_PORT_80_TCP_PORT': '80', 'PATH': '/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/intel/compilers_and_libraries/linux/mpi/bin64', 'MINICONDA_VERSION': '4.7.12.1', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_PORT_80_TCP': 'tcp://10.0.202.194:80', 'PWD': '/var/azureml-app', 'AZUREML_ENVIRONMENT_IMAGE': 'True', 'AZUREML_CONDA_ENVIRONMENT_PATH': '/azureml-envs/azureml_037efdbdc846c714c9ae62fc8e6d162d', 'LANG': 'C.UTF-8', 'SHLVL': '0', 'HOME': '/root', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'PYTHONPATH': '/var/azureml-server:', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_SERVICE_HOST': '10.0.202.194', 'SVDIR': '/var/runit', 'DEBIAN_FRONTEND': 'noninteractive', 'CAAS_4D736229D6AB4F719927FCFB196F10B4_PORT_80_TCP_ADDR': '10.0.202.194', 'KUBERNETES_PORT_443_TCP_ADDR': '10.0.0.1', 'KUBERNETES_PORT_443_TCP': 'tcp://10.0.0.1:443', 'SERVER_SOFTWARE': 'gunicorn/19.9.0', 'PYSPARK_SUBMIT_ARGS': 'pyspark-shell', 'KMP_INIT_AT_FORK': 'FALSE'}

[vizhur]

Environment.environment_variables are runtime variables, that is a bit confusing :(.

Documentation is pretty clear though, variables are not a part of container image but will be available for execution

https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.environment.environment?view=azure-ml-py

[chengyu-liu-cs]

@vizhur Could you provide more information if you also feel it is confusing ? Not just copy paste the document which I already checked and didnot feel it provides any valuable information ? The script could be anything that a user write, for example a scoring file is also a script. Please provide more precise.

Based on the materials, Environment image will be built automatically from Environment instance, then shouldn't these variables defined inside Environment object be inside the environment docker image built ?Can you at least try to explain where these environment_variables are stored and how they are extracted when executed ? These details are really important for me and our company to design solutions for ourselves and our customers

[chengyu-liu-cs]

Can anyone reopen this ?

[vizhur]

@chengyu-liu-cs sorry for delayed response. Sorry to hear that our documentation does not provide any valuable information to you. The confusion I meant is that Environment object has some runtime properties that we can't deprecate with 1.x version for backward compatibility. Regarding environment variables, these are not a part of the docker image and will be set by execution for docker runs. By execution I mean the service that performs experiment.submit()., for any config that implements RunConfiguration in most of the cases that would be ScriptRunConfiguration. So when run happens on that docker image these variables will be set for the process. I hope I answered you question and once again sorry for confusion and your bad experience.

Please follow up our release notes for any updates and improvements.

[chengyu-liu-cs]

@vizhur Thank for the kind reply. So then is there a plan to include these environment variables in docker image not use them as runtime properties?

[vizhur]

@chengyu-liu-cs there are no current plans to support that for 1.x SDK unfortunately. Please follow our release notes for the updates https://docs.microsoft.com/en-us/azure/machine-learning/azure-machine-learning-release-notes