Things we can test for automatically

[jmandel]

... this will be a long list, but for now, just the stuff we notice on manual inspection that we can/should automate tests for. Much of this should be done by off-the-shelf testing tools (we should confirm).

• Format of references (make sure absolute URL or relative FHIR URL)
• Check search results Bundle.type=searchset

[erikwiffin]

There are tests for everything in the list so far. Keeping this as an open issue so that we can add test ideas without too much overhead.

[jmandel]

OAuth testing for state -- make sure it's correctly returned (unmodified). Including some simple state ("123") and also some "tricky" state (like values that need URL encoding).

[jmandel]

(Probably not automatically, but maybe for our "security checklist": is CSRF protection present on the "approve" screen.)

[jmandel]

Automated login via webdriver to move beyond the "Authorize" button. Maybe initially it's still an authorize button but an automatic one.

[jmandel]

Including the wrong value for redirect_uri

• at the authorize step
• at the token step.

Ensure that servers fail on wrong grant_type on authorize

Servers should fail when presented with the same code twice (i.e. repeated attempts to call /token should fail).

[jmandel]

Instead of Python validator: POST to $validate instead :-)

[jmandel]

Check vocabularies. For each LOINC code, SNOMED code, and RxNorm code in a FHIR payload:

• ensure that the code meets value set expectations, if any (e.g. RxNorm SCD, SBD, GPCK, BPCK for medications, and IN or BN for allergies)

These tests would be "optional" in the sense that they're informative; we wouldn't fail vendors for low scores here, but we'd provide some high-level feedback about quality of coding.

[jmandel]

Make sure that, somewhere, the Bundle.link.next works. For example:

• On laboratory-results, follow the link once, or to completion, and fail on broken links.