Define and prioritize critical user journeys

[dcmcand]

Context

Currently this project has no long running integration tests that reflect critical user journeys. This means that we can't have confidence that a release will not break critical workflows when the system is deployed. It currently contains comprehensive unit tests which test functions in isolation, and short integration tests which test subsystems in relatively straightforward ways. These short tests are excellent for a rapid feedback loop for developers, but fall short of comprehensively testing critical user paths in the application.

I propose defining the critical user paths that we feel need to be tested to ensure that future releases do not contain bugs that would prevent users from using conda-store and prioritizing them by the highest value. We can then ticket out writing the individual integration tests. This will iteratively improve the release process until we have high confidence in the stability of releases.

This testing should occur at the api level rather than the UI level. This greatly simplifies writing tests while fully testing the critical paths. API-level testing is typically more stable and less prone to changes, making it a robust foundation for ensuring core functionality works as expected. Separate UI integration tests can be written that test only the logic contained in frontend code. These long running integration tests would also be limited to running on release creation to keep feedback loops short for developers.

Please contribute use user journeys that you think are important to test before a release. High level descriptions are fine for an initial pass, we can refine these when we ticket them individually.

Value and/or benefit

• Long-running integration tests that cover critical user journeys can significantly boost confidence in the stability of software releases. By simulating real-world user interactions, you can identify issues that might not be caught by unit tests or short integration tests.
• Identifying and addressing issues in critical user paths before a release can help prevent critical bugs from making their way into production releases. This reduces the likelihood of disrupting users' workflows and lowers the cost of fixing issues later in the development cycle.
• Ensuring that critical user workflows are thoroughly tested can result in a better user experience. Users are less likely to encounter frustrating and show-stopping bugs when using the software, leading to higher satisfaction and increased adoption.

Anything else?

No response

[trallard]

@kcpevey

[kcpevey]

@dcmcand I'm sure there are other workflows, but these are the most common ones that come to mind. I hope its what you are looking for.

At first I thought I'd cover the steps, but then I realized that the steps are part of the UI, not the backend so I didn't bother with continuing that. That said, at some point, we should be testing these workflows on the frontend as well.

Workflow 1 Persona: I'm a software eng familiar with conda cli tools. Need: I want to create a new env via the conda-store-ui (on Nebari, e.g.). Steps:

• Log into conda-store
• click plus to start a new env in my user space (breakout workflow: group user space)
• create environment via ui widgets (breakout workflow: use yaml editor)
• Click "Create"
• I should be redirected back to the "non-edit" screen where it says its "Building"

Workflow 2 Persona: I'm a tech lead that managed envs for my team Need: I want to delete an environment

Workflow 3 Persona: I'm a sys admin for Nebari and I need to free up space on the NFS Need: I want to delete old environment builds

Workflow 4 Persona: I'm a tech lead that managed envs for my team Need: I need to change which build is selected as "Active" for this environment.

Workflow 4 Persona: I'm a software engineer who manages my own envs Need: I have an env that failed to build, I need to investigate the logs to determine the cause of the failure.

Workflow 5 Persona: I'm a software engineer developing a package Need: I need to get a lockfile (breakout workflow: pinned yaml file or docker image) to store for ensured reproducibility

Workflow 6 Persona: I'm a software engineer who manages my own envs Need: I just realized I added something dumb to a huge environment and just submitted it to build. I want to cancel this build rather than wait an hour for it to complete and rebuild.

[costrouc]

@nkaretnikov comment on this issue with stories.