Possible missing VirtualService for the dashboard-links relation

[AnotherStranger]

Bug Description

The mlflow charm fails to provide a working link to the mlflow UI in the Kubeflow dashboard. After deploying and relating the charms, the link is created but it does not work (possibly due to a missing VirtualService). The juju debug logs only contain some warnings on a missing refresh_event (This seems to be unrelated to this report).

@kimwnasptd and I already had a short discussion in the matrix chatroom about this.

If there is anything I can do to assist you further please let me know.

Here is a screenshot of the dashboard link which gets created after enabling the relation:

To Reproduce

1. Deploy mlflow using channel 2.1/stable with juju deploy mlflow --channel=2.1/stable --trust
2. Deploy resource-dispatcher using channel 1.0/stable with juju deploy resource-dispatcher --channel=1.0/stable --trust
3. Relate the charms: mlflow-server:pod-defaults to resource-dispatcher:pod-defaults, and mlflow-server:secrets to resource-dispatcher:secrets
4. Relate the charms again: mlflow-server:dashboard-links to kubeflow-dashboard:links

Environment

• Platform: Charmed Kubeflow using channel 1.8/stable on Microk8s
• Juju version: Channel 3.1/stable

Relevant Log Output

application-minio: 10:06:17 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
unit-metacontroller-operator-0: 10:07:13 WARNING unit.metacontroller-operator/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
application-envoy: 10:07:18 WARNING unit.envoy/3.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
unit-mlflow-server-0: 10:07:42 WARNING unit.mlflow-server/0.juju-log 2 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
application-mlflow-minio: 10:09:23 WARNING unit.mlflow-minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.

Additional Context

No response

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5880.

This message was autogenerated

[kimwnasptd]

Thanks for opening the issue @AnotherStranger!

You could help us initially by confirming the VirtualService that we need to ensure the access is there, and then we can figure out how to integrate it into the Charm.

It should look very similar to the other ones, like this one but change the destination to be the K8s SVC that is created for MLflow via the Charm.

Most probably we might need to play with the rewrite of the URL as well. Could you give it a spin?

[AnotherStranger]

Hey @kimwnasptd,

I got access to the mlflow app using following VirtualService (Adapted from your linked one):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: mlflow-server-mlflow-server
spec:
  gateways:
  - kubeflow-gateway
  hosts:
  - '*'
  http:
  - headers:
      request:
        add:
          x-forwarded-prefix: /mlflow
    match:
    - uri:
        prefix: /mlflow/
    rewrite:
      uri: /
    route:
    - destination:
        host: mlflow-server
        port:
          number: 5000

[kimwnasptd]

Nice, the last step now will be to extend the charm to also create this VirtualService.

There are some details to be discussed though on how this should be done, since the rest of the charms are using the ingress relation and SDI (which we want to move away from) https://github.com/canonical/notebook-operators/blob/track/1.8/charms/jupyter-ui/src/charm.py#L420-L429

[misohu]

We already have relation with istio to create the virtual service

juju relate istio-pilot:ingress mlflow-server:ingress

Can you try it please?

[AnotherStranger]

We already have relation with istio to create the virtual service

juju relate istio-pilot:ingress mlflow-server:ingress

Can you try it please?

Sorry that I didn't get back to you sooner. The relation adds the correct VirtualService. Thank you!

[DnPlas]

Hey @AnotherStranger after creating the correct VirtualService, were you able to see the MLFlow component in the dashboard?

[AnotherStranger]

Hey @AnotherStranger after creating the correct VirtualService, were you able to see the MLFlow component in the dashboard?

Oh, yes! I should have clarified this in my response. with the istio-pilot relation everything works as expected.

[DnPlas]

Awesome, thank you very much for confirming! I will be closing this issue as it is not required anymore, feel free to re-open or file a new one if needed.