Open cesarvarela opened 5 months ago
I would propose we develop probes and data tests. These are slightly more descriptive than "smoke" tests for our purposes.
Probes would test the accessibility of features on the live production AIID site using Playwright. These include: landing page, review queue, risk checklists, news digest, and other dynamic features that rely on external APIs.
Data tests could strictly query the MongoDB with assertions about the data. For example, no null
fields on incident objects (e.g. https://github.com/responsible-ai-collaborative/aiid/pull/3104), or all incidents have reports, etc.
Should we then rename current playwright folders as:
e2e-full -> e2e e2e -> probes
And create a new data
folder.
For the probe test creation, starting with the following urls:
https://incidentdatabase.ai/ https://incidentdatabase.ai/apps/submitted/ https://incidentdatabase.ai/apps/newsdigest/ https://staging-aiid.netlify.app/apps/checklists/
@cesarvarela and @kepae
In commit: https://github.com/scottslewis/aiid/commit/199ddaeeea49efa38ebc8af0aedc99d394fd66aa
I've moved probes.ts to a new directory (e2e/probes/) and refactored the tests so that there is one const 2-dim array that defines the tests and now have only one test method. That test method now iterates through the array to call testUrl (urlPrefix, paths) for each url prefix/paths combination defined in urls1.
Feel free to make any desired stylistic or doc changes.
It should be hosted in a
smoke
folder next to thee2e
folder.Playwright docs show an example implementation using projects:
https://playwright.dev/docs/test-projects#splitting-tests-into-projects