Closed martin-weber closed 4 years ago
The pipeline steps are running in a Docker container (mcr.microsoft.com/mlops/python:latest) which has all packages installed from ci_dependecies.yml
Thanks @eedorenko . If I understand that right, I need to create a custom container if I want to add packages in ci_dependencies.yml and have them ready for my unit tests. Is that right?
So far I did not use a custom container as in conda_dependencies.yml the comment says:
# Conda environment specification. The dependencies defined in this file will
# be automatically provisioned for managed runs. These include runs against
# the localdocker, remotedocker, and cluster compute targets.
Does this mean that the dependencies defined here should be available by the CI unit test? I added lightgbm in my conda_dpendencies.yml (see below) and in the ci_dependecies.yml (see below) but do not find it in the conda and pip packages (see logs below).
conda_dpendencies.yml
name: mlopsohmw_training_env
dependencies:
# The python interpreter version.
# Currently Azure ML Workbench only supports 3.5.2 and later.
- python=3.7.*
- pip
- pip:
# Base AzureML SDK
- azureml-sdk==1.3.*
# Minimum required for the scoring environment. Must match AzureML SDK version.
# https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments
- azureml-defaults==1.3.*
# Training deps
- numpy==1.18.*
- pandas==1.0.*
- scikit-learn==0.22.*
- lightgbm==2.3.*
# Scoring deps
- inference-schema[numpy-support]
# MLOps with R
- azure-storage-blob
# current project
- azureml-dataprep
- azureml-monitoring
ci_dependencies.yml
name: mlopspython_ci
dependencies:
# The python interpreter version.
- python=3.7.*
# dependencies with versions aligned with conda_dependencies.yml.
- numpy=1.18.*
- pandas=1.0.*
- scikit-learn=0.22.*
- lightgbm=2.3.*
# dependencies for MLOps with R.
- r=3.6.0
- r-essentials=3.6.0
- pip=20.0.*
- pip:
# dependencies with versions aligned with conda_dependencies.yml.
- azureml-sdk==1.3.*
# Additional pip dependencies for the CI environment.
- pytest==5.4.*
- pytest-cov==2.8.*
- requests==2.23.*
- python-dotenv==0.12.*
- flake8==3.7.*
- flake8_formatter_junit_xml==0.0.*
- azure-cli==2.3.*
- lightgbm==2.3.*
I added a step to show the environment at the beginning of .pipelines\code-quality-template.yml
- bash: |
conda info -e
conda list
pip list
displayName: 'show environment'
These are the outputs:
It is also confusing why the base conda environment is active. conda info -e
# conda environments:
#
base * /usr/local
mlopspython_ci /usr/local/envs/mlopspython_ci
conda list
# packages in environment at /usr/local:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
asn1crypto 1.3.0 py37_0
ca-certificates 2020.1.1 0
certifi 2020.4.5.1 py37_0
cffi 1.14.0 py37h2e261b9_0
chardet 3.0.4 py37_1003
conda 4.8.3 py37_0
conda-env 2.6.0 1
conda-package-handling 1.6.0 py37h7b6447c_0
cryptography 2.8 py37h1ba5d50_0
idna 2.9 py_1
ld_impl_linux-64 2.33.1 h53a641e_7
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libstdcxx-ng 8.2.0 hdf63c60_1
ncurses 6.2 he6710b0_0
openssl 1.1.1f h7b6447c_0
pip 20.0.2 py37_1
pycosat 0.6.3 py37h7b6447c_0
pycparser 2.20 py_0
pyopenssl 19.1.0 py37_0
pysocks 1.7.1 py37_0
python 3.7.5 h0371630_0
readline 7.0 h7b6447c_5
requests 2.23.0 py37_0
ruamel_yaml 0.15.87 py37h7b6447c_0
setuptools 46.1.3 py37_0
six 1.14.0 py37_0
sqlite 3.31.1 h7b6447c_0
tk 8.6.8 hbc83047_0
tqdm 4.44.1 py_0
urllib3 1.25.8 py37_0
wheel 0.34.2 py37_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zlib 1.2.11 h7b6447c_3
pip list
Package Version
------------------------------------- -------------------
adal 1.2.2
antlr4-python3-runtime 4.8
applicationinsights 0.11.9
argcomplete 1.11.1
attrs 19.3.0
azure-batch 8.0.0
azure-cli 2.3.1
azure-cli-command-modules-nspkg 2.0.3
azure-cli-core 2.3.1
azure-cli-nspkg 3.0.4
azure-cli-telemetry 1.0.4
azure-common 1.1.25
azure-core 1.4.0
azure-cosmos 3.1.2
azure-datalake-store 0.0.48
azure-functions-devops-build 0.0.22
azure-graphrbac 0.60.0
azure-identity 1.2.0
azure-keyvault 1.1.0
azure-loganalytics 0.1.0
azure-mgmt-advisor 2.0.1
azure-mgmt-apimanagement 0.1.0
azure-mgmt-appconfiguration 0.4.0
azure-mgmt-applicationinsights 0.1.1
azure-mgmt-authorization 0.52.0
azure-mgmt-batch 7.0.0
azure-mgmt-batchai 2.0.0
azure-mgmt-billing 0.2.0
azure-mgmt-botservice 0.2.0
azure-mgmt-cdn 4.1.0rc1
azure-mgmt-cognitiveservices 5.0.0
azure-mgmt-compute 12.0.0
azure-mgmt-consumption 2.0.0
azure-mgmt-containerinstance 1.5.0
azure-mgmt-containerregistry 3.0.0rc11
azure-mgmt-containerservice 9.0.1
azure-mgmt-cosmosdb 0.12.0
azure-mgmt-datalake-analytics 0.2.1
azure-mgmt-datalake-nspkg 3.0.1
azure-mgmt-datalake-store 0.5.0
azure-mgmt-datamigration 0.1.0
azure-mgmt-deploymentmanager 0.2.0
azure-mgmt-devtestlabs 2.2.0
azure-mgmt-dns 2.1.0
azure-mgmt-eventgrid 2.2.0
azure-mgmt-eventhub 3.0.0
azure-mgmt-hdinsight 1.4.0
azure-mgmt-imagebuilder 0.2.1
azure-mgmt-iotcentral 3.0.0
azure-mgmt-iothub 0.11.0
azure-mgmt-iothubprovisioningservices 0.2.0
azure-mgmt-keyvault 2.2.0
azure-mgmt-kusto 0.3.0
azure-mgmt-loganalytics 0.3.0
azure-mgmt-managedservices 1.0.0
azure-mgmt-managementgroups 0.2.0
azure-mgmt-maps 0.1.0
azure-mgmt-marketplaceordering 0.2.1
azure-mgmt-media 1.1.1
azure-mgmt-monitor 0.7.0
azure-mgmt-msi 0.2.0
azure-mgmt-netapp 0.8.0
azure-mgmt-network 10.0.0
azure-mgmt-nspkg 3.0.2
azure-mgmt-policyinsights 0.4.0
azure-mgmt-privatedns 0.1.0
azure-mgmt-rdbms 2.2.0
azure-mgmt-recoveryservices 0.4.0
azure-mgmt-recoveryservicesbackup 0.6.0
azure-mgmt-redis 7.0.0rc1
azure-mgmt-relay 0.1.0
azure-mgmt-reservations 0.6.0
azure-mgmt-resource 8.0.1
azure-mgmt-search 2.1.0
azure-mgmt-security 0.1.0
azure-mgmt-servicebus 0.6.0
azure-mgmt-servicefabric 0.4.0
azure-mgmt-signalr 0.3.0
azure-mgmt-sql 0.17.0
azure-mgmt-sqlvirtualmachine 0.5.0
azure-mgmt-storage 8.0.0
azure-mgmt-trafficmanager 0.51.0
azure-mgmt-web 0.44.0
azure-multiapi-storage 0.2.4
azure-nspkg 3.0.2
azure-storage-blob 1.5.0
azure-storage-common 1.4.2
azureml-automl-core 1.3.0
azureml-core 1.3.0
azureml-dataprep 1.4.3
azureml-dataprep-native 14.1.0
azureml-pipeline 1.3.0
azureml-pipeline-core 1.3.0
azureml-pipeline-steps 1.3.0
azureml-sdk 1.3.0
azureml-telemetry 1.3.0
azureml-train 1.3.0
azureml-train-automl-client 1.3.0
azureml-train-core 1.3.0
azureml-train-restclients-hyperdrive 1.3.0
backcall 0.1.0
backports.tempfile 1.0
backports.weakref 1.0.post1
bcrypt 3.1.7
bleach 3.1.4
certifi 2020.4.5.1
cffi 1.14.0
chardet 3.0.4
cloudpickle 1.3.0
colorama 0.4.3
contextlib2 0.6.0.post1
coverage 5.1
cryptography 2.9
decorator 4.4.2
defusedxml 0.6.0
distro 1.5.0
docker 4.2.0
dotnetcore2 2.1.13
entrypoints 0.3
fabric 2.5.0
flake8 3.7.9
flake8-formatter-junit-xml 0.0.6
fusepy 3.0.1
humanfriendly 8.1
idna 2.9
importlib-metadata 1.5.0
invoke 1.4.1
ipykernel 5.1.4
ipython 7.13.0
ipython-genutils 0.2.0
isodate 0.6.0
javaproperties 0.5.1
jedi 0.16.0
jeepney 0.4.3
Jinja2 2.11.1
jmespath 0.9.5
joblib 0.14.1
jsmin 2.2.2
jsondiff 1.2.0
jsonpickle 1.4
jsonschema 3.2.0
junit-xml 1.9
jupyter-client 6.1.2
jupyter-core 4.6.3
knack 0.7.0rc1
MarkupSafe 1.1.1
mccabe 0.6.1
mistune 0.8.4
mkl-fft 1.0.15
mkl-random 1.1.0
mkl-service 2.3.0
mock 4.0.2
more-itertools 8.2.0
msal 1.2.0
msal-extensions 0.1.3
msrest 0.6.13
msrestazure 0.6.3
nbconvert 5.6.1
nbformat 5.0.4
ndg-httpsclient 0.5.1
notebook 6.0.3
numpy 1.18.1
oauthlib 3.1.0
packaging 20.3
pandas 1.0.3
pandocfilters 1.4.2
paramiko 2.7.1
parso 0.6.2
pathspec 0.8.0
pexpect 4.8.0
pickleshare 0.7.5
pip 20.0.2
pkginfo 1.5.0.1
pluggy 0.13.1
portalocker 1.7.0
prometheus-client 0.7.1
prompt-toolkit 3.0.4
ptyprocess 0.6.0
py 1.8.1
pyasn1 0.4.8
pycodestyle 2.5.0
pycparser 2.20
pyflakes 2.1.1
Pygments 2.6.1
PyJWT 1.7.1
PyNaCl 1.3.0
pyOpenSSL 19.1.0
pyparsing 2.4.7
pyrsistent 0.16.0
pytest 5.4.1
pytest-cov 2.8.1
python-dateutil 2.8.1
python-dotenv 0.12.0
pytz 2019.1
PyYAML 5.3.1
pyzmq 17.1.2
requests 2.23.0
requests-oauthlib 1.3.0
ruamel.yaml 0.15.89
scikit-learn 0.22.1
scipy 1.4.1
scp 0.13.2
SecretStorage 3.1.2
Send2Trash 1.5.0
setuptools 46.1.3.post20200330
six 1.14.0
sshtunnel 0.1.5
tabulate 0.8.7
terminado 0.8.3
testpath 0.4.4
tornado 6.0.4
traitlets 4.3.3
urllib3 1.25.8
vsts 0.1.25
vsts-cd-manager 1.0.2
wcwidth 0.1.9
webencodings 0.5.1
websocket-client 0.56.0
wheel 0.30.0
xmltodict 0.12.0
zipp 2.2.0
Yes, @martin-weber, you understand that right.
Thanks @eedorenko. Ok that solves my issue.
It looks as the step Run unit tests in Model CI Pipeline has not yet the packages from ci_dependecies.yml or conda_dependencies.yml applied. If I get it right, the conda packages are first installed later by step the Publish Azure Machine Learning Pipeline when creating the environment.
What is the best way to have the same modules/packages installed for the unit tests and for the later Environment?