VirtualService and stuff deployed with InferenceService gets overridden

[A923357]

Hello everyone,

I'm working on creating an OAuth 2.0 service using Kubeflow, Istio, and KServe. As part of this setup, I need to add a custom response header to my Istio VirtualService, which is managed by a KServe InferenceService. Additionally, I've configured an Istio EnvoyFilter to handle invalid tokens by converting 302 redirects into appropriate 40X error codes.

Initially, everything works perfectly—the custom header is added, and invalid tokens are handled correctly without causing 302 redirects. However, after a few hours, I notice that:

1. My VirtualService configuration is being overridden, removing the custom response header.
2. The system reverts to returning 302 redirects instead of the intended 40X error codes.
3. The service stops working as expected due to these changes.
4. I've read in some forums that this issue might be related to the Knative-KServe reconciliation process, where Knative periodically reconciles resources and may overwrite custom configurations in Istio resources like VirtualService. I'm not entirely certain if this is the cause, but it seems plausible given the symptoms.

My questions are:

• Is Knative-KServe reconciliation the reason why my VirtualService configuration is being overridden?

• If so, how can I prevent Knative from overwriting my custom VirtualService configurations, such as adding a custom response header and maintaining the EnvoyFilter settings?

• Are there best practices or recommended approaches to persist custom Istio configurations when using Knative and KServe?

I've also posted this issue in the KServe repository but wanted to reach out here for additional insights and assistance.

Any guidance or suggestions on how to resolve this issue would be greatly appreciated!

Thank you for your help.

- knative-serving version: 1.8.1

[skonto]

Hi @A923357 I recommend that you ask first at the KServe side. The virtual service Knative touches is the one it generates, cc @ReToCode in case I missed something. Btw I suspect you could debug this by using raw deployments at the KServe side (instead of serverless mode) and see how things work the reconciliation. In any case pls ask Kserve community providing mode details about the resources touched.

[ReToCode]

+1 on asking the KServe folks.