⚗️ ♻️ improving e2e tests (adding structure for additional CI tests)

matusdrobuliak66 commented 1 week ago

What do these changes do?

As part of the system tests, we are already running e2e-playwright tests.
Currently (thanks to Sylvain's great work), these tests are executed in the CI whenever someone creates a pull request. They are implemented as follows:
- make install-ci-up-simcore. This command:
- Starts the local registry.
- Pulls and pushes the sleeper image to the local registry.
- Runs the entire SimCore stack.
- Grants service access rights in the database.
- Then we run make test-sleepers
My current improvements are:
- Additionally, we run make test-platform.
- Here, we can add additional small frontend tests that we want to include as part of the CI process.
- NOTE: I will modify these tests, when osparc-testing-id will be added on the new UI elements (folders and workspaces) -> UPDATE: tests modified based on these IDs https://github.com/ITISFoundation/osparc-simcore/pull/6766
- The test is designed to reuse the same browser context, so you only need to log in/out once, which speeds things up.
- My idea is that, with this approach, it will force us to prepare both setup and teardown in the code (similar to unit tests). One of the next steps could be to add an SQL file to inject data for these tests (NOTE: this file will need to be maintained).

Disclaimer: This is just an experiment. If we find that it is not useful, we do not need to continue. However, I would like to give it a try.

Related issue/s

How to test

Dev-ops checklist

[ ] No ENV changes or I properly updated ENV (read the instruction)

codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.53%. Comparing base (793318d) to head (7566445). Report is 1 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #6765 +/- ## ========================================== + Coverage 88.11% 88.53% +0.42% ========================================== Files 1512 1542 +30 Lines 60601 62208 +1607 Branches 2125 2125 ========================================== + Hits 53398 55078 +1680 + Misses 6875 6802 -73 Partials 328 328 ``` | [Flag](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [integrationtests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `64.87% <ø> (+11.35%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `86.46% <ø> (+0.12%)` | :arrow_up: | | [Components](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | Coverage Δ | | |---|---|---| | [api](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [pkg_aws_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `93.38% <ø> (ø)` | | | [pkg_dask_task_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `96.87% <ø> (ø)` | | | [pkg_models_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `91.93% <ø> (ø)` | | | [pkg_notifications_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `83.79% <ø> (ø)` | | | [pkg_postgres_database](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `87.33% <ø> (ø)` | | | [pkg_service_integration](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `71.44% <ø> (ø)` | | | [pkg_service_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `76.48% <ø> (ø)` | | | [pkg_settings_library](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `91.42% <ø> (ø)` | | | [pkg_simcore_sdk](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.27% <ø> (ø)` | | | [agent](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `96.98% <ø> (ø)` | | | [api_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.88% <ø> (ø)` | | | [autoscaling](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `95.22% <ø> (ø)` | | | [catalog](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.42% <ø> (ø)` | | | [clusters_keeper](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `98.72% <ø> (ø)` | | | [dask_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `91.32% <ø> (ø)` | | | [datcore_adapter](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `94.05% <ø> (ø)` | | | [director](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `75.99% <ø> (ø)` | | | [director_v2](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.91% <ø> (ø)` | | | [dynamic_scheduler](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `96.59% <ø> (ø)` | | | [dynamic_sidecar](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.78% <ø> (ø)` | | | [efs_guardian](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.01% <ø> (ø)` | | | [invitations](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `93.51% <ø> (ø)` | | | [osparc_gateway_server](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `85.15% <ø> (ø)` | | | [payments](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `92.87% <ø> (ø)` | | | [resource_usage_tracker](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `90.80% <ø> (+0.14%)` | :arrow_up: | | [storage](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `89.76% <ø> (∅)` | | | [webclient](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `∅ <ø> (∅)` | | | [webserver](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) | `88.75% <ø> (+1.16%)` | :arrow_up: | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Last update [793318d...7566445](https://app.codecov.io/gh/ITISFoundation/osparc-simcore/pull/6765?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ITISFoundation).

🚨 Try these New Features: