FHIR endpoints do not consider dailybuild content

[danka74]

I have a snowstorm instance with dailybuild loaded. The native API allows looking up unreleased content, e.g. http://localhost:8080/MAIN/SNOMEDCT-SE/concepts/65081000052104 where as the FHIR API throws a Concept not found exception: http://localhost:8080/fhir/CodeSystem/$lookup?system=http://snomed.info/sct/45991000052106&code=65081000052104

Possibly, this could be something you might not want to fix... /Daniel

[kaicode]

Yes, the FHIR endpoints use the latest published content as a conscious decision. I wonder if there is a way to specify unpublished content in FHIR..

[danka74]

...but this works: http://localhost:8080/fhir/CodeSystem/$lookup?system=http://snomed.info/sct/45991000052106/version/20200531&code=65081000052104

[kaicode]

Right, Snowstorm defaults to content of the latest code system version where the effective time is not in the future. You can change this behaviour in configuration using the codesystem.all.latest-version.allow-future property.

[pgwilliams]

Yes it was a decision made here to only serve up the latest published version. The code originally looked at the MAIN branch or the equivalent.

I was thinking about specifying a version greater than those present in the release but that seems a bit haphazard. A flag in the config file perhaps to indicate use of the daily build branch by default?

[danka74]

Makes sense, I'll try the property. /Daniel

[danka74]

FHIR endpoints above give the same results as without the property set for $expand but works without version in URL for $lookup. Tried this (which worked on 4.8.0...): http://localhost:8080/fhir/ValueSet/$expand?url=http://snomed.info/sct/45991000052106/version/20200531?fhir_vs=ecl/%5E64691000052109&count=10&designation=sv which still gives no results even though snowstorm is in the right branch: 2020-04-17 16:04:35.690 INFO 16878 --- [nio-8080-exec-4] o.s.s.f.services.FHIRValueSetProvider : Recovered: 0 concepts from branch: MAIN/SNOMEDCT-SE/2020-05-31 with ecl: '^64691000052109' for comparison: http://localhost:8080/MAIN%2FSNOMEDCT-SE/members?referenceSet=64691000052109&offset=0&limit=50 gives the expected results.

[pgwilliams]

Oh the FHIR Code doesn't know about that property @kaicode has mentioned, I'd need to write code for it to respect that. The question is whether we want to use that one and have an effective_time in the future, or have a separate configuration item to indicated use of MAIN/ ie the daily build by default when a version parameter is not specified.

Well since we've already got a solution for this on the snowowl API side I guess it makes sense to re-use it. So if you agree (@kaicode @danka74 ) I'll say that any effective time in the future goes to the Daily Build branch? But only when that property is set to true.

[danka74]

Not sure what the impact of the alternatives is, and what amount of work is needed, but I would trust you to do what you find best! Thanks, Daniel

[kaicode]

From what I understand there are two use cases here:

• Selecting future dated but versioned content
• Selecting content which may not be versioned and for which there is no code system version entry

We only have a solution for the former as far as I know.

@pgwilliams The config property I mentioned changes how the code system "latestVersion" is selected. It can control if a future dated version can be used as the latest version.

I expect that to work in the FHIR code naturally using this line:

class FHIRHelper {
...
    public String getBranchPathForCodeSystemVersion(StringType codeSystemVersionUri) {
...
        // Lookup latest effective version (future versions will not be used until publication date)
        branchPath = codeSystem.getLatestVersion().getBranchPath();

https://github.com/IHTSDO/snowstorm/blob/4.8.0/src/main/java/org/snomed/snowstorm/fhir/services/FHIRHelper.java#L97

Exposing unversioned content on MAIN or other extension main branches via FHIR seems prone to misinterpretation. We should probably only allow that if it is explicitly requested somehow?

[danka74]

Hi @kaicode , I have imported the dailybuild as well as refsets with "createCodeSystemVersion": true and that should take care of versioning the imported content, shouldn't it?

[kaicode]

@danka74 no, I don't think so because the effectiveTime is not set in the daily build archives - although don't know about the refsets rows. The Snowstorm daily build process does not create (and recreate daily) a code system version because it doesn't expect the content to have an effective time. I know you have not been able to use the dailybuild import function (I have asked Terance for a copy of your S3 credentials).

If you imported manually and set that flag I'm not sure what will happen. If the refset rows had a date set it's possible that was used and a code system version created but then we are not talking about a daily build as we understand it - this is more like importing a release.

[danka74]

The refset is imported separately from the daily build. The way I have managed to perform a ValueSet expansion now is to:

1. create (i.e. PUT) a ValueSet resource including the following compose:

   compose: {
   include: [
   {
   system: "http://snomed.info/sct/45991000052106/version/20200531",
   filter: [
   {
   property: "constraint",
   op: "=",
   value: "^64691000052109"
   }
   ]
   }
   ]
   },

2. import 20200309 international
3. import 20191130 SE extension
4. import refset backdated to 20191130
5. import dailybuild delta with createCodeSystemVersion: true (not sure if this matters or not). Now I can expand the ValueSet: http://localhost:8080/fhir/ValueSet/reason-for-encounter/$expand and get the ValueSet back including 20200531 content. Still, http://localhost:8080/fhir/ValueSet/$expand?url=http://snomed.info/sct/45991000052106?fhir_vs=ecl/%5E64691000052109&displayLanguage=sv&count=10 gives nothing back.

[pgwilliams]

Unfortunately the FHIR spec doesn't have any notion of wanting to work with unversioned unpublished content and it does seem like a reasonable thing to want to do on an instance being used to create new content as @danka74 is doing.

As you say @kaicode what we've been discussing above is open to misinterpretation and using a future effectiveTime is a) not what was intended for that flag and b) would give the impression of success if a server was not kept up to date and a client legitimately requested some new published release which wasn't loaded onto the server.

We discussed a solution which we might get away with of using a known string value as the version eg http://localhost:8080/fhir/CodeSystem/$lookup?system=http://snomed.info/sct/45991000052106/version/UNVERSIONED&code=65081000052104

That seems to be working with the current build. I've got logging set at debug level locally and I get this output when I use this parameter value: 2020-04-20 16:50:32.459 DEBUG 69151 --- [nio-8080-exec-3] o.s.snowstorm.fhir.services.FHIRHelper : Request to use unversioned content, using daily build branchpath: MAIN/SNOMEDCT-SE

...which will also work with the expand operation, so you should be able to work with those new reference sets you're adding @danka74 . Expand example: http://localhost:8080/fhir/ValueSet/$expand?url=http://snomed.info/sct/45991000052106/version/UNVERSIONED?fhir_vs=isa/27624003&designation=sv

[danka74]

Thanks Peter, I can confirm that it works, but also that referring to future releases .../version/20200531 works. Could this be because i imported the daily build with createCodeSystemVersion: true? E.g. http://localhost:8080/fhir/ValueSet/$expand?url=http://snomed.info/sct/45991000052106/version/20200531?fhir_vs=ecl/%5E64691000052109&displayLanguage=sv&count=10 works fine now where it used not to. /Daniel

[pgwilliams]

Yes, I'm pretty sure you'll have created a codesystem version for that date. You can look at all versions for a codesystem using something like:
https://browser.ihtsdotools.org/snowstorm/snomed-ct/codesystems/SNOMEDCT-SE/versions?showFutureVersions=true