SeldonIO / seldon-core

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
https://www.seldon.io/tech/products/core/
Other
4.38k stars 831 forks source link

mlflow server fail on conda env create #4378

Closed jneo8 closed 2 years ago

jneo8 commented 2 years ago

Describe the bug

When deploy a mlflow model with seldonio/mlflowserver:1.14.0-dev. It will fail when the classifier init the conda environment. Which is wired because the conda.yaml is pretty simple.

Defaulted container "classifier" out of: classifier, seldon-container-engine, istio-proxy, classifier-model-initializer (init), istio-init (init)
Executing before-run script
---> Creating environment with Conda...
INFO:root:Copying contents of /mnt/models to local
INFO:root:Reading MLmodel file
INFO:root:Creating Conda environment 'mlflow' from conda.yaml
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
/opt/conda/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)

ResolvePackageNotFound: 
  - python=3.7.15

Traceback (most recent call last):
  File "./conda_env_create.py", line 151, in <module>
    main(args)
  File "./conda_env_create.py", line 146, in main
    setup_env(model_folder)
  File "./conda_env_create.py", line 54, in setup_env
    create_env(env_file_path)
  File "./conda_env_create.py", line 110, in create_env
    run(cmd, shell=True, check=True)
  File "/opt/conda/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'conda env create -n mlflow --file /tmp/tmp94xo5f9p/conda.yaml' returned non-zero exit status 1.

seldondeployment.yaml

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: mlflow
spec:
  name: wines
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - name: classifier
          image: seldonio/mlflowserver:1.14.0-dev
          imagePullPolicy: Always
          livenessProbe:
            initialDelaySeconds: 80
            failureThreshold: 200
            periodSeconds: 5
            successThreshold: 1
            httpGet:
              path: /health/ping
              port: http
              scheme: HTTP
          readinessProbe:
            initialDelaySeconds: 80
            failureThreshold: 200
            periodSeconds: 5
            successThreshold: 1
            httpGet:
              path: /health/ping
              port: http
              scheme: HTTP
    graph:
      children: []
      implementation: MLFLOW_SERVER
      modelUri: ${MODEL_URI}
      envSecretRefName: seldon-init-container-secret
      name: classifier
    name: wine-super-model
    replicas: 1

conda.yaml

channels:
- conda-forge
dependencies:
- python=3.7.15
- pip<=22.0.4
- pip:
  - mlflow
  - cloudpickle==2.2.0
  - scikit-learn==1.0.2
name: mlflow-env

Environment

microk8s 1.24

axsaucedo commented 2 years ago

Closing as PR has been merged

gpotti commented 1 year ago

Could someone please advise which seldonio/mlserver image has this fix included? I tried 1.15.1 and the same issue exists.