search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
102 stars 49 forks source link

Unable to create Experiment from Dashboard #375

Closed RosaCandela closed 2 years ago

RosaCandela commented 3 years ago

/kind bug

What steps did you take and what happened:

We installed Kubeflow from Microk8s on a Ubuntu VM, using the following commands:

We accessed the dashboard using the link http://10.64.140.43.nip.io/, displayed at the end of the installation, and enabling the SOCKS proxy.

Once logged in the dashboard, we are unable to create an experiment and, consequently a run, from a pipeline. This is the error we get when clicking on Create experiment:

Experiment creation failed {"error":"Validate experiment request failed.: Invalid input error: In single-user mode, CreateExperimentRequest shouldn't contain resource references.","code":3,"message":"Validate experiment request failed.: Invalid input error: In single-user mode, CreateExperimentRequest shouldn't contain resource references.","details":[{"@type":"type.googleapis.com/api.Error","error_message":"In single-user mode, CreateExperimentRequest shouldn't contain resource references.","error_details":"Validate experiment request failed.: Invalid input error: In single-user mode, CreateExperimentRequest shouldn't contain resource references."}]}

What did you expect to happen: How to create an experiment or run a pipeline

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

Pedrohgv commented 3 years ago

Having the same issue. Any solution for this yet?

G-Lauz commented 3 years ago

Hi! I have the same issue but with a different error:

What steps did you take and what happened:

I installed Kubeflow on Microk8s with Charmed Kubeflow using the followings commands:

$ sudo snap install microk8s --classic
$ sudo usermod -a -G microk8s $USER
$ su - $USER

$ microk8s.enable dns storage ingress istio
$ microk8s.enable metallb:10.64.140.43-10.64.140.49

$ sudo snap install juju --classic
$ juju bootstrap microk8s micro
$ juju add-model kubeflow
$ juju deploy cs:kubeflow-lite

$ microk8s.kubectl patch role -n kubeflow istio-ingressgateway-operator -p '{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"name":"istio-ingressgateway-operator"},"rules":[{"apiGroups":["*"],"resources":["*"],"verbs":["*"]}]}'

$ juju config dex-auth public-url=http://<istio-ingressgateway address>.nip.io
$ juju config oidc-gatekeeper public-url=http://<istio-ingressgateway address>.nip.io

I can access the dashboard with the same address define in the last commands e.g. http://10.64.140.44.nip.io

Once logged in the dashboard, I'm unable to create an experiment and, consequently a run, from a pipeline. This is the error I get when clicking on Create experiment:

An error occurred
{"error":"Invalid input error: In single-user mode, ListExperiment cannot filter by namespace.","code":3,"message":"Invalid input error: In single-user mode, ListExperiment cannot filter by namespace.","details":[{"@type":"type.googleapis.com/api.Error","error_message":"In single-user mode, ListExperiment cannot filter by namespace.","error_details":"Invalid input error: In single-user mode, ListExperiment cannot filter by namespace."}]}

It's also the same error that were report in this issue: #393

What did you expect to happen:

Create an experiment and then run a pipeline.

Anything else you would like to add:

I also tried the patches from @phvalguima in these issues: #393 and #394 and I end up with the same result as him.

Environment:

phvalguima commented 3 years ago

@G-Lauz @RosaCandela I've eventually exposed the kfp-ui service directly, via NodePort as a work-around:

$ kubectl expose deployment -n kubeflow --type=NodePort kfp-ui --name=kfp-ui-nodeport
$ kubectl describe svc -n kubeflow kfp-ui-nodeport # Find out which port is it mapped on the node

Then, you can connect to the kfp-ui-nodeport directly: ip-of-your-host:nodeport. That works for me.

As pointed out below, that means your kfp-ui-nodeport will be available without authentication in this case.

G-Lauz commented 3 years ago

Thanks you for your answer @phvalguima.

Your solution work well for me too. But I'm concern about the security issue that can lead to.

By pushing a bit your idea I found out that the error come up only when we aren't using a namespace (e.g. : admin (owner)) and using the default namespace instead. If I manage to not login through the Dex authentication page I can access the dashboard in that default namespace and then everything work well (except that I can't access any notebook because we need a namespace for them).

The issue might be relate to this kubeflow/pipeline#1881

wodecki commented 2 years ago

Solved by reinstalling kubeflow according to instructions provided here: https://charmed-kubeflow.io/docs/quickstart?_ga=2.139718198.46250420.1643883037-872003253.1643883037

DnPlas commented 2 years ago

Hi @RosaCandela, this is a known issue present in our previous release that is fixed in our latest (v1.4). The Kubeflow add-on for Microk8s is an old version and won't get updates in the short or mid-term, we recommend you deploy Kubeflow following the instructions in our Install guide instead of using the add-on. I'm closing this issue, but feel free to re-open or file a new one in case you come across any problem.