responsible-ai-collaborative / aiid

The AI Incident Database seeks to identify, define, and catalog artificial intelligence incidents.
https://incidentdatabase.ai
Other
172 stars 35 forks source link

Create a smoke + live probe + data consistency test suite #2886

Open cesarvarela opened 5 months ago

cesarvarela commented 5 months ago

It should be hosted in a smoke folder next to the e2e folder.

Playwright docs show an example implementation using projects:

https://playwright.dev/docs/test-projects#splitting-tests-into-projects

kepae commented 2 months ago

I would propose we develop probes and data tests. These are slightly more descriptive than "smoke" tests for our purposes.

Probes would test the accessibility of features on the live production AIID site using Playwright. These include: landing page, review queue, risk checklists, news digest, and other dynamic features that rely on external APIs.

Data tests could strictly query the MongoDB with assertions about the data. For example, no null fields on incident objects (e.g. https://github.com/responsible-ai-collaborative/aiid/pull/3104), or all incidents have reports, etc.

cesarvarela commented 1 month ago

Should we then rename current playwright folders as:

e2e-full -> e2e e2e -> probes

And create a new data folder.

scottslewis commented 1 month ago

For the probe test creation, starting with the following urls:

https://incidentdatabase.ai/ https://incidentdatabase.ai/apps/submitted/ https://incidentdatabase.ai/apps/newsdigest/ https://staging-aiid.netlify.app/apps/checklists/

scottslewis commented 1 month ago

@cesarvarela and @kepae

In commit: https://github.com/scottslewis/aiid/commit/199ddaeeea49efa38ebc8af0aedc99d394fd66aa

I've moved probes.ts to a new directory (e2e/probes/) and refactored the tests so that there is one const 2-dim array that defines the tests and now have only one test method. That test method now iterates through the array to call testUrl (urlPrefix, paths) for each url prefix/paths combination defined in urls1.

Feel free to make any desired stylistic or doc changes.