[Bug]: Istio fails to run when installed with ODH on IBM Cloud cluster

opendatahub-io / opendatahub-community

Apache License 2.0

26 stars 34 forks source link

[Bug]: Istio fails to run when installed with ODH on IBM Cloud cluster #148

Open eyalcha opened 9 months ago

eyalcha commented 9 months ago

ODH Component

ODH Operator

Current Behavior

Istio Ingress / Egress gateways fails on IBM Cloud cluster. If I patched it with runAsNonRoot: false all runs well.

Expected Behavior

Services should run

Steps To Reproduce

Install ODH 2.8.0 with servicemesh

Workaround (if any)

patched it with runAsNonRoot: false

What browsers are you seeing the problem on? (If applicable)

Chrome

Open Data Hub Version

2.8.0

Anything else

zdtsw commented 8 months ago

I would not think this is a bug rather a different config in the cluster which not work with the standard offering of ODH.

plus, use Root to run container might bring some security concern.

jiridanek commented 8 months ago

Random observation, I noticed service-ca operator has special manifest for deployment on IBM Cloud, https://github.com/openshift/service-ca-operator/blob/master/manifests/05_deploy-ibm-cloud-managed.yaml.

The same project recently configured the required-scc

https://github.com/openshift/service-ca-operator/pull/235/files

Anyways, this issue is with https://github.com/maistra/istio-operator, if I am not mistaken. The gateways in the report are pods in the istio-system namespace, named something like istio-egressgateway-7c46668687-r8lzs, istio-ingressgateway-77f94d8f85-vzsq9.

@eyalcha Does the issue with the istio gateway pods startup/scheduling appear right after you install the servicemesh operator, even before you even install ODH on IBM Cloud?

What is the precise error message you are seeing? You haven't quoted a single message from the system in your bug report. Do you happen to see the Error: container has runAsNonRoot and image will run as root by any chance?

eyalcha commented 7 months ago

@jiridanek I think the servicemesh now installed after installing ODH operator as part of the DCI. In any case, it happens after servicemesh is installed for the first time. I don't have the precise error message now, will try to recreate the error.

jiridanek commented 7 months ago

I got myself IBM cloud. Looking into community-operators catalogsource, I do see ODH, but I can't find version 2.8.0 there, only 2.10. I have OpenShift 4.15, maybe that's why, and I'd need 4.14 to be able to install older ODH.

The 2.10 version seems to have been installed correctly. From DSC status

  installedComponents:
    codeflare: false
    kserve: true
    trustyai: false
    ray: false
    kueue: false
    data-science-pipelines-operator: true
    workbenches: true
    model-registry-operator: false
    model-mesh: true
    dashboard: true
  phase: Ready

jiridanek commented 6 months ago

@eyalcha any updates?

zdtsw commented 2 months ago

@eyalcha should we close this issue?