inferno-framework / uds-plus-test-kit

Conformance Testing for the UDS+ Implementation Gudie
Apache License 2.0
3 stars 3 forks source link

Fix faulty URL and block CPT code checking while running locally #17

Closed ljtucker closed 10 months ago

ljtucker commented 10 months ago

Summary

Corrected two existing issues:

@arscan please take a look when you get a chance. Thank you!

arscan commented 10 months ago

Thanks @ljtucker. Do you happen to have a public example of a code that isn't working so we can replicate? There shouldn't be a difference in running this locally, so I have a feeling there is a bug somewhere that we'd like track down in the IG/validator/tx.fhir.org for a global fix. But your workaround here looks pretty straightfoward and isolated.

ljtucker commented 10 months ago

@arscan Here is an example resource that was triggering the error: https://github.com/drajer-health/uds-plus/blob/master/input/examples/Encounter-de-identified-example.json

It is not a difference in code, only the information the validator service has access to. The hosted server run by your team is providing a list of CPT codes to check against, access to which is a paid service. When running locally, the validator service does not have access to that list, and therefore can't validate the codes, triggering the error.

arscan commented 10 months ago

Thanks @ljtucker. Note that even when you are using a local validator, it is still configured by default to use tx.fhir.org for terminology validation, which is the same service that the hosted validator on inferno.healthit.gov points to for non-certification related tests (including this one).

We think it might be a caching issue, or maybe an issue with what IGs are loaded on inferno.healthit.gov's validator vs. your local validator (if you remember that can cause some weird behavior). Also, the fact that the CPT code recently changed potentially might be an issue (regarding caching).

Can you provide the exact error(s) in your local instance here just so we can rule out a few things? Just filtering out this specific CPT error is an option to make the error go away, but we'd also like to figure out why this is happening.

ljtucker commented 10 months ago

image

@arscan here is the screenshot we received from the customer. I can confirm that, as of last Wednesday, the given CPT code could be validated by the hosted server, but not locally. If it was a caching issue exclusively, then an encounter with that cpt code should pass locally without any other code changes once the cache is refreshed. I cannot provide you that exact resource, but I believe the resource I did send you has a similar CPT code attached. Please let me know if there is anything else I can do to help test the caching problem before this PR gets merged. Thank you!

ljtucker commented 10 months ago

Hi @arscan we are getting requests from users for the fix of the faulty URL. What is the timeline for your investigation into the caching issue? No rush on that fix, so if you think it will take time to resolve, let me know and I will create a separate PR for just the fix to the URL. Thanks!

ljtucker commented 10 months ago

@arscan Thank you for getting it merged! Please keep me updated on if/when you find a long-term solution to the bug. I would like to remove that error blocker if we can be sure the test is accurate in the future. I'll go ahead and write an issue ticket as a reminder.

arscan commented 10 months ago

Note that we are scheduled for our next deployment to inferno.healthit.gov early next week. Let me know that if that profile URL fix is important enough to expedite though.