canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
104 stars 50 forks source link

Seldon Core does not work behind the proxy #548

Open Barteus opened 1 year ago

Barteus commented 1 year ago

When deploying the Seldon Core behind the proxy it gets stuck on creating a Conda environment. The same container without Istio Pods works fine.

The model was build using e2e-wine notebook from examples and stored in Minio to eliminate the problem with setting proxy for initializer.

cat <<EOF | kubectl apply -n admin -f -
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
 name: mlflow
spec:
 name: wines
 predictors:
 - componentSpecs:
   - spec:
       containers:
       - name: classifier
         image: seldonio/mlflowserver:1.14.0-dev
         livenessProbe:
           initialDelaySeconds: 80
           failureThreshold: 200
           periodSeconds: 5
           successThreshold: 1
           httpGet:
             path: /health/ping
             port: http
             scheme: HTTP
         readinessProbe:
           initialDelaySeconds: 80
           failureThreshold: 200
           periodSeconds: 5
           successThreshold: 1
           httpGet:
             path: /health/ping
             port: http
             scheme: HTTP
         env:
         - name: HTTP_PROXY
           value: <proxy>
         - name: http_proxy
           value: <proxy>
         - name: HTTPS_PROXY
           value: <proxy>
         - name: https_proxy
           value: <proxy>
         - name: NO_PROXY
           value: <no_proxy>
         - name: no_proxy
           value: <no_proxy>
       - name
   graph:
     children: []
     implementation: MLFLOW_SERVER
     modelUri: s3://mlflow/5/853a874c47364a46bbdf9c979bc935ec/artifacts/model
     envSecretRefName: seldon-init-container-secret
     name: classifier
   name: wine-super-model
   replicas: 1
EOF

Log from classifier:

Executing before-run script
---> Creating environment with Conda...
INFO:root:Copying contents of /mnt/models to local
INFO:root:Reading MLmodel file
INFO:root:Creating Conda environment 'mlflow' from conda.yaml
Collecting package metadata (repodata.json): ...working... failed
/opt/conda/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)

ProxyError: Conda cannot proceed due to an error in your proxy configuration.
Check for typos and other configuration errors in any '.netrc' file in your home directory,
any environment variables ending in '_PROXY', and any other system-wide proxy
configuration settings.

As a part of the experiments I have deployed only the classifier container and it works fine with this configuration.

Barteus commented 1 year ago

I have added Istio config and with env properties for proxy.

kubectl apply -n kubeflow -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: proxy
spec:
  hosts:
  - my-company-proxy.com # ignored
  addresses:
  - <proxy-ip>/32
  ports:
  - number: <proxy-port>
    name: tcp
    protocol: TCP
  location: MESH_EXTERNAL
EOF

It works now, but I'm not sure how upgrades etc. will be handled or if charm overrides my config.