Private Endpoint Tasks

[Jooho]

This is tracking GH issue.

• odh-model-controller (https://github.com/opendatahub-io/odh-model-controller/pull/221, https://github.com/opendatahub-io/odh-model-controller/pull/229)

  • annotate serving cert for runtime service and copy it to istio-system namespace.
  • update gateway.
  • Sync to 0.12.0 (https://github.com/opendatahub-io/odh-model-controller/pull/231)

• upstream/kserve

  • create a new Gateway/Service in doc
  • update VS local gateway name (https://github.com/kserve/kserve/pull/3737)

• opendatahub-operator (https://github.com/opendatahub-io/opendatahub-operator/pull/1056)

  • create a new Gateway/Service for kserve

• odh/kserve

  • update inferenceservice configmap
    • master - https://github.com/opendatahub-io/kserve/pull/372
    • 0.12.1 - https://github.com/opendatahub-io/kserve/pull/373
  • cherry-pick creating VS PR
    • master - https://github.com/opendatahub-io/kserve/pull/369
    • 0.12.1 - https://github.com/opendatahub-io/kserve/pull/370)
  • update VS local gateway name (https://github.com/opendatahub-io/kserve/pull/376)

[Jooho]

Full test

Setup Test Environment to verify all PRs PR list

• kserve - 0.12.1 branch
  • Allow KServe to have its own local gateways for Serverless mode
  • update local gateway information
  • Prevent the PassthroughCluster for clients/workloads in the service mesh (kserve#3711)
• opendatahub operator - incubating branch
  • [RHOAIENG-7919] add kserve-local-gateway Gateway and Service
• odh model controller - master branch
  • Feature: Create cert Secret and update KServe local gateway

images

• quay.io/jooholee/odh-model-controller:private
• quay.io/jooholee/kserve-controller:private
• quay.io/jooholee/opendatahub-operator:private
• quay.io/jooholee/opendatahub-operator-catalog:0.0.0

Manual way

Install custom odh operator/kserve operator/odh-model-controller operator

• Manifests

  • CUSTOM_KSERVE_MANIFESTS=https://github.com/jooho/kserve/tarball/test-manifest
  • CUSTOM_ODH_MODEL_CONTROLLER_MANIFESTS=https://github.com/jooho/odh-model-controller/tarball/test-manifests
  • CUSTOM_ODH_OPERATOR_CATALOGSOURCE_IMG=quay.io/jooholee/opendatahub-operator-catalog:v0.0.0

• Deploy test sklearn iris model

  • ServingRuntime: https://raw.githubusercontent.com/Jooho/loopy/main/src/roles/kserve/samples/sklearn/manifests/sklearn/sklearn-runtime.yaml
  • ISVC: https://raw.githubusercontent.com/Jooho/loopy/main/src/roles/kserve/samples/sklearn/manifests/sklearn/sklearn-isvc-iris-v2-storageuri-rest.yaml

---

Script way

Loopy Setup refer this doc

git clone   git@github.com:Jooho/loopy.git
cd loopy
make init

or

docker run -it quay.io/jooholee/loopy

cat cluster_info.sh
CLUSTER_CONSOLE_URL=https://console-openshift-console.XXXX
CLUSTER_API_URL=https://api.XXX:443
CLUSTER_ADMIN_ID=admin
CLUSTER_ADMIN_PW=password
CLUSTER_TOKEN=sha256~XXX
CLUSTER_TYPE=ROSA

Install Kserve with a new manifests

loopy playbooks run private-endpoint-poc -p CUSTOM_KSERVE_MANIFESTS=https://github.com/jooho/kserve/tarball/test-manifest  -p CUSTOM_ODH_MODEL_CONTROLLER_MANIFESTS=https://github.com/jooho/odh-model-controller/tarball/test-manifests -vvv -i ./cluster_info.sh

Deploy sample sklearn model

loopy units run test-kserve-sklearn-v2-iris-role -vvv -i ./cluster_info.sh 

---

Test from the same namespace

1. Running test pods

   
   TEST_NAMESPACE=kserve-demo

oc label namespace ${TEST_NAMESPACE} pod-security.kubernetes.io/enforce=baseline --overwrite oc label namespace ${TEST_NAMESPACE} pod-security.kubernetes.io/warn=baseline --overwrite oc label namespace ${TEST_NAMESPACE} pod-security.kubernetes.io/audit=baseline --overwrite

oc run test-with-istio -n ${TEST_NAMESPACE} --image=registry.access.redhat.com/rhel7/rhel-tools --annotations=sidecar.istio.io/inject="true" -- sleep infinity 2> /dev/null oc run test -n ${TEST_NAMESPACE} --image=registry.access.redhat.com/rhel7/rhel-tools -- sleep infinity 2> /dev/null

2. curl from a pod with istio-proxy from the same cluster

oc exec test-with-istio -n ${TEST_NAMESPACE} -- curl -s sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live

3. curl from a pod without istio-proxy from the same cluster

oc exec test -n ${TEST_NAMESPACE} -- curl -sk https://sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live

## Test from another namespace

1. Running test pods

ANOTHER_NAMESPACE=curl-test oc new-project ${ANOTHER_NAMESPACE} oc label namespace ${ANOTHER_NAMESPACE} pod-security.kubernetes.io/enforce=baseline --overwrite oc label namespace ${ANOTHER_NAMESPACE} pod-security.kubernetes.io/warn=baseline --overwrite oc label namespace ${ANOTHER_NAMESPACE} pod-security.kubernetes.io/audit=baseline --overwrite cat << EOF |oc create -f - kind: ServiceMeshMember apiVersion: maistra.io/v1 metadata: name: default namespace: ${ANOTHER_NAMESPACE} spec: controlPlaneRef: name: data-science-smcp namespace: istio-system EOF

oc run test-with-istio -n ${ANOTHER_NAMESPACE} --image=registry.access.redhat.com/rhel7/rhel-tools -n ${ANOTHER_NAMESPACE} --annotations=sidecar.istio.io/inject="true" -- sleep infinity 2> /dev/null oc run test -n ${ANOTHER_NAMESPACE} --image=registry.access.redhat.com/rhel7/rhel-tools -n ${ANOTHER_NAMESPACE} -- sleep infinity 2> /dev/null

2. curl from a pod with istio-proxy from another cluster

oc exec test-with-istio -n ${ANOTHER_NAMESPACE} -- curl -s sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live

3. curl from a pod without istio-proxy from another cluster

oc exec test -n ${ANOTHER_NAMESPACE} -- curl -sk https://sklearn-example-isvc-iris-v2-rest.${TEST_NAMESPACE}.svc.cluster.local/v2/health/live