Supporting simple user story

[rokroskar]

This is a meta-issue to keep track of work done to support the fourth sprint of supporting the basic data science user story.

Please add a link to the relevant issue once it's created in the corresponding repository. Track the fine-grained issues in sprint project.

0. Housekeeping

• [ ] document the ontology for projects, datasets, and CWL steps/workflows
  • [ ] https://github.com/SwissDataScienceCenter/renga-ontology/issues/3, see also the data model described in the incubator UI
• [x] Travis-ci integration test for the development branch of renga (https://github.com/SwissDataScienceCenter/renga/pull/139)
• [x] add storage to docker-compose (#128)
• [x] document how the storage service should be used with other components (@ebiiii)
• [x] Finalize staging / CD (https://github.com/SwissDataScienceCenter/testing.datascience.ch/pull/5)
• [x] set up demo script (see https://github.com/SwissDataScienceCenter/renga-demo)

1. Initiate a project, import a dataset

• [ ] Finalize dataset import to work with renga repos (https://github.com/SwissDataScienceCenter/renga-python/issues/99)
• [x] Support creating a project from the UI

2. Explore the dataset using a notebook (isolated environment)

3. Run code on data, recording data provenance for reproducibility

• [ ] POC for extending renga run to more flexible environments
  • [ ] a dockerized version of renga run/rerun
  • [ ] using python environments in steps rerun on the server (à la repo2docker)
• [ ] write user stories as test cases for renga run (https://github.com/SwissDataScienceCenter/renga-python/issues/151)

4. Push project to a server for sharing

• [x] Finalize storage service
  • [x] decouple from graph (https://github.com/SwissDataScienceCenter/renga-storage/issues/44)
  • [x] update swagger and doc (https://github.com/SwissDataScienceCenter/renga-storage/issues/50)
  • [x] define bucket/backend for LFS (https://github.com/SwissDataScienceCenter/renga-storage/issues/51)
• [x] Add an endpoint in the storage to download an object by its checksum (https://github.com/SwissDataScienceCenter/renga-storage/issues/54)
• [ ] Manipulate project access settings in the UI (https://github.com/SwissDataScienceCenter/renga-ui/issues/130)
• [x] Initialize a project in UI (templates? Gitlab-ci, .renga metadata etc.; https://github.com/SwissDataScienceCenter/renga-ui/issues/129)

5. Validate results/workflows

• [x] document the available options for executing CWL workflows on k8s -> https://docs.google.com/document/d/1mUPlU64vpRDHMPmYoUoJieOPdMrwQXYXwMDZfdL9amw

6. Open discussions about the project

https://github.com/SwissDataScienceCenter/renga-ui/issues/132

Stylistic issues:

• [x] Tabs or pills for card navigation [Bootstrap Card Nav]
• [x] Support creating projects (#129)
• [x] Improve the layout of the projects page ( #141)
• [x] Improve the layout of the Kus page ( #142)
• [ ] Ku properties should be displayed in lists and details page (#113)
• [x] Lineage of files should be displayed UI/159
• [ ] Improve styling of Markdown content to be better integrated with UI
• [ ] Improve styling of notebook content to be better integrated with UI
• [x] Show Avatars instead of generic icons for users

7. Provide a runnable notebook for exploring new ideas

• [x] POC for using jupyterhub with CI stages for deploying notebooks (#147)
• [x] enable jupyterhub integration in k8s (#152)
• [x] enable notebooks to run with custom images (#176)
• [ ] build images on push if repo includes a Dockerfile

8. Rerun workflows with new data

9. Visualize lineage

• [x] provide lineage collected offline to UI for visualization #160 interpret the lineage information in the repo and generate events consume lineage events and produce a view to be revisited once we have an event-queue/graph in place

[rokroskar]

closing this issue for now -- continued as https://github.com/SwissDataScienceCenter/renku/issues/235 with more specific issues related to our internal tutorial day