End-to-end test coverage

[kschiffer]

Summary

This umbrella issue keeps track of our end-to-end test coverage.

• [ ] Applications
  • [x] CRUD #3650
  • [x] Checking overview (information displayed as expected) #3663
  • [x] Payload formatters (set all types, remove) #3841
  • [ ] Integrations
    • [ ] MQTT (Check credentials)
    • [x] Webhooks (CRUD, with and without webhook template)
    • [x] Pub/sub (CRUD) #3720
  • [x] Collaborators (CRUD) #3443, #3646
  • [x] API Keys (CRUD) #3452
• [x] End devices
  • [x] CRUD
    • [x] Create
    • [x] Read
    • [x] Update
    • [x] Delete
  • [x] Import
  • [x] Checking overview (information displayed as expected) #3663
  • [x] Scheduling downlinks #3883
  • [x] Location (setting and removing) @asmulko on this atm
  • [x] Payload formatters (set types, remove, check precedence over app level PFs) #3841
  • [x] Claim authentication code (setting and removing)
  • [x] General Settings (test various sub-forms, removing gateway)
• [ ] Gateways
  • [x] CRUD #3672
  • [x] Checking overview (information displayed as expected) #3663
  • [x] Location (setting and removing) #3681
    • [ ] Confirm updating via status message
  • [x] Collaborators (CRUD) #3443, #3646
  • [x] API Keys (CRUD) #3452
• [x] Organizations
  • [x] CRUD #3670
  • [x] Checking overview (information displayed as expected) #3663
  • [x] Collaborators (CRUD) #3443, #3646
  • [x] API Keys (CRUD) #3452
• [x] Authentication and authorization
  • [x] Login
  • [x] Logout (Console and OAuth app)
  • [x] User registration
  • [x] Change password (when authenticated)
  • [x] Change password (using temporary password)
  • [x] Authorizing an OAuth client
  • [x] Verifying contact info
  • [x] Retrieving authorization code (/oauth/code)
• [ ] Miscellaneous
  • [ ] Feature toggles
    • [ ] Check various rights setups and confirm:
      • [ ] Correct visibility of items
      • [ ] Correct write restriction of items
  • [ ] Data / Event streams (all entities)
    • [ ] we need some work to enabling simulating events from within the cypress test #6694
    • [ ] Confirm correct rendering of various event types
    • [ ] Test pausing and clearing
  • [ ] Error pages
  • [ ] User management (CRUD)
  • [x] Search functionality in tables
  • [ ] Some smoke testing of responsive styles
• [ ] Add functional tests to storybook #3478
• [ ] Add page title assertions to e2e tests #3479
• [x] Speed up e2e execution in CI #3480
• [ ] Add clipboard support for e2e tests #3481
• [x] Add direct database access for end-to-end tests #3605
• [x] Save screenshots as artifacts for failed e2e tests #3523
• [x] Packet broker
• [ ] MAC Settings

Also note the complementary umbrella issue for enterprise features: https://github.com/TheThingsIndustries/lorawan-stack/issues/2575

Everyone please keep this up to date and add test-cases that you think are important, or split up cases into more detailed sub-cases.

For everyone that wants to dive into writing end-to-end tests, please see the guide: https://github.com/TheThingsNetwork/lorawan-stack/blob/v3.10/DEVELOPMENT.md#end-to-end-tests

[kschiffer]

I'll add the rest of the dev team as assignees once #3611 is done.

[asmulko]

@kschiffer I assume that Application / Linking e2e tests can be removed from the list, because it will be removed in v3.11, see #3614 and we can put Payload formatters on hold as well

[kschiffer]

Yes indeed. Linking can be removed, though Payload formatters will stay untouched as far as I understand.

[johanstokking]

What is the status here? What can we improve on the short term still?

[kschiffer]

@asmulko made a lot of progress with this recently and I think the coverage is pretty good already. The most important thing short-term is traffic simulation, which will enable us putting a whole range of critical end-to-end tests in place. This is also tracked in #3611. Then we need to start looking into performance optimizations #3480.

[johanstokking]

OK. Should we track end-to-end testing in this issue or in #3611, or is there a reason to keep both?

[kschiffer]

I'd prefer keeping and maintaining this overview here as well. We can update this issue with a lot more test cases and onboard more of the team to write end-to-end tests after #3611 is done.

[kschiffer]

Little reminder for me, @bafonins, and @ryaplots to keep the list above updated and to periodically check back whether some of those can be picked up by us.

[kschiffer]

I think we an close this now. The rest of the missing coverage is either tracked by individual issues or not planned currently.