search

fhir-crucible / crucible

🔥 Crucible web application for testing FHIR servers
77 stars 25 forks source link

Supported Only Filter not working? #207

Closed jawalonoski closed 7 years ago

jawalonoski commented 7 years ago

See https://chat.fhir.org/#narrow/stream/crucible/subject/.E2.80.9Csupported.20only.E2.80.9D.20filter

okeefm commented 7 years ago

Do we filter nightly runs by supported tests only?

jimsteel commented 7 years ago

The servers I was referring to in the chat are https://projectcrucible.org/servers/582f8e486372757e2e000000 (FHIR 1.8.0) and https://projectcrucible.org/servers/57d30d486372756382090000 (FHIR 1.4.0)

arscan commented 7 years ago

Thanks for reporting this @jimsteel. The problem is that our automated tests that run twice weekly do not obey the 'supported only' filter. I believe this was intentional, in case the server did not report a proper conformance. But this causes bad behavior in servers like yours, because you are getting a much lower 'grade' than you should because your server only claims to support a subset of the spec. So unless @okeefm or @jawalonoski have an issue with me changing this, I'll go ahead and update the nightly runs so that they only run tests that are supported by each server.

It'll take at least a few days to get this up into production, and to fix your history so that the unsupported test runs are erased. I'll update this issue when this is complete.

Thanks!