E2E test for jupyter web app

jlewi commented 5 years ago

We should have E2E tests for the Jupyter web app using a framework like selenium.

See kubeflow/testing#288

issue-label-bot[bot] commented 5 years ago

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.95. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

kimwnasptd commented 5 years ago

I can try and take that one.

I'll start with some basic end-to-end cases such as creating and deleting Notebooks and will gradually try to add more advanced ones.

/assign kimwnasptd

jlewi commented 5 years ago

@kimwnasptd any update on this?

jlewi commented 4 years ago

Bump to P0 because this is a requirement for 1.0.

kimwnasptd commented 4 years ago

@jlewi I'm reading the e2e instructions in the testing repo and I have some questions.

Does each test (.py file) need to be encapsulated in an argo workflow in order to be triggered from prow?
- If yes, where should this bundle (.py + argo app) live in the code base?
- Will the argo workflow also need to be added to the repo's prow_config.yaml to be run as a postsubmit job?
The test code will need to have access to a kubeflow cluster with the latest jupyter-web-app image deployed. How will it know this endpoint? I see the testing#273 but I'm not sure of its current state.
Will the test code be responsible for setting up the kubeflow cluster or is this automated and a kubeflow endpoint could be provided as an ENV Var (for example) in the container that will run the test?

jlewi commented 4 years ago

Does each test (.py file) need to be encapsulated in an argo workflow in order to be triggered from prow? If yes, where should this bundle (.py + argo app) live in the code base?

An E2E test will be an Argo workflow that contains steps for building and deploying the application and then testing it. The invocation of the python code (e.g. pytests) to test your application will be steps in this workflow.

The code for the primary E2E test for kubeflow is defined in kfctl_e2e_workflow.py
So you can use that as the basis for defining an E2E workflow for the jupyter web app
If you want to create a presubmit test that you would write an E2E workflow that would have Argo steps for the following
- Build the jupyter web app
- Modify the manifests to use your new image
- Deploy Kubeflow using the updated image
- Run one or my pytests to verify the jupyter web app is working
You will like want to define this workflow in the same repository as the source code; i.e. kubeflow/klubeflow
- Python code should go under py/kubeflow/${REPO} to allow consistent absolute imports
- So you'd probably put it in py/kubeflow/kubeflow/ci
Once you have a python file you can add it to the prow file.
Writing a postsubmit test might be a little easier because you could run the test against the most recently deployed cluster from master. In which case you wouldn't have to worry updating the manifests or deploying the Kubeflow clusters.
- You can model this on create_e2e_workflow.py which is a test for our examples
  - In this case the test has a step to get credentials to talk to the latest cluster deployed from master

It might be useful to discuss at the community meeting or in an out of band meeting.

jlewi commented 4 years ago

@kimwnasptd any update on this?

jlewi commented 4 years ago

@kimwnasptd any update on this?

fediazgon commented 4 years ago

If @kimwnasptd is Ok with it I can start working on this. I've seen there are a lot of area/jupyter tickets in KF1.0 that could be easily tested if we have a suite in place.

I've already started to refactor https://github.com/jlewi/kubeflow/blob/1690703d261d29ccd7e1d5ff154d79064473d9d8/kubeflow/jupyter/tests/jupyter_test.py#L85

kimwnasptd commented 4 years ago

Just finished the basic structure of the test. It has two test cases for now:

Check that the index page can load and doesn't have a popup error
Check that the form page can load and doesn't have a popup error

Lets first add these tests to have something to work with and we can then further expand them.

@jlewi IIUC the python test file will be in the kubeflow/kfctl repo, under the py/kubeflow/kfctl/testing/pytests/ directory correct?

jlewi commented 4 years ago

@kimwnasptd You could probably put it in kubeflow/kfctl or kubeflow/kubeflow depending on what is easier. The e2e test will be split across those repos since it depends on code in multiple repos.

jlewi commented 4 years ago

@kimwnasptd Any update on this? Do you have a link to your PR?

@fediazgon I think what you are working on is complementary. https://github.com/jlewi/kubeflow/blob/1690703d261d29ccd7e1d5ff154d79064473d9d8/kubeflow/jupyter/tests/jupyter_test.py#L85 is testing that Jupyter itself is accessible; i.e. its testing that the notebook controller correctly sets up ISTIO reverse proxy routes for notebooks.

I think @kimwnasptd is testing the UI for launching notebooks (i.e. creating Notebook resources).

I think we need both. So if you want to go ahead and work on testing the Jupyter Notebook UI that would be a huge help. Perhaps we should open up a separate issue for that?

fediazgon commented 4 years ago

@kimwnasptd Any update on this? Do you have a link to your PR?

@fediazgon I think what you are working on is complementary. https://github.com/jlewi/kubeflow/blob/1690703d261d29ccd7e1d5ff154d79064473d9d8/kubeflow/jupyter/tests/jupyter_test.py#L85 is testing that Jupyter itself is accessible; i.e. its testing that the notebook controller correctly sets up ISTIO reverse proxy routes for notebooks.

I think @kimwnasptd is testing the UI for launching notebooks (i.e. creating Notebook resources).

I think we need both. So if you want to go ahead and work on testing the Jupyter Notebook UI that would be a huge help. Perhaps we should open up a separate issue for that?

I was working on the Argo workflow that you mentioned in this comment to wrap @kimwnasptd tests. My idea was to add a flag in create_unique_kf_instance to allow creating a deployment but overriding some images. Then, test this deployment in the pytests.

jlewi commented 4 years ago

@fediazgon replied in slack; my suggestion would be to start by running in postsubmits and periodic tests run against our auto-deployed clusters so that we don't have to modify create_unique_kf_instance. I added some pointers in slack. We can continue to discuss there.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jlewi commented 4 years ago

@kimwnasptd What's the status of E2E testing for the jupyter web app? Are we running and reporting the e2e test results?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jlewi commented 4 years ago

@kimwnasptd and @StefanoFioravanzo any update on this?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

jlewi commented 3 years ago

@kubeflow/wg-notebook-leads Any update on this?

thesuperzapper commented 3 years ago

I will raise this in the notebook working group slack channel.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

kimwnasptd commented 3 years ago

/reopen /lifecycle frozen

google-oss-robot commented 3 years ago

@kimwnasptd: Reopened this issue.

In response to [this](https://github.com/kubeflow/notebooks/issues/70): >/reopen >/lifecycle frozen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

juliusvonkohout commented 1 week ago

/transfer notebooks

juliusvonkohout commented 1 day ago

/close as decided by @andreyvelich @tarilabs in the migration document

google-oss-prow[bot] commented 1 day ago

@juliusvonkohout: Closing this issue.

In response to [this](https://github.com/kubeflow/notebooks/issues/70#issuecomment-2468351185): >/close >as decided by @andreyvelich @tarilabs in the migration document Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubeflow / notebooks

E2E test for jupyter web app #70