davidmiller commented 6 years ago

Our current default symptoms lookuplist has 85 entries. Can we find Snomed codes for them all, and update the data file here: https://github.com/openhealthcare/opal/blob/v0.11.0/opal/core/referencedata/data/lookuplists/symptom.json to include them in the JSON with the syntax:

   {
            "name": "Lumbar Epidural",
            "synonyms": ["Insertion of temporary catheter into lumbar epidural space (procedure)", "Epidural for Labour"],
            "coding": {
                "code": "708911002",
                "system": "http://snomed.info/sct"
             }
        }

This allows us to move forward with writing tests for https://github.com/openhealthcare/opal/compare/terminology and implementing it to have a single proof of concept snomed coded lookuplist.

Adding additional keys to the lookuplist data format should be backwards compatible as the loader will simply ignore them.

ayamane commented 6 years ago

I can take this on but would need to know what you want for the synonyms? Do you want to limit it to a certain number of items? e.g. if we query SNOMED using "Abdominal Swelling" there are 7 matching terms that are returned: http://browser.ihtsdotools.org/api/v2/snomed/en-edition/v20180131/descriptions?query=Abdominal%20Swelling

davidmiller commented 6 years ago

Hi @ayamane !

We'd appreciate the help :)

This is of course very much a judgement call, and we'd likely want this reviewed by someone with a clinical background (no idea if this describes you or not, if not then we can def. find someone).

I don't think I would look to limit by number of items in the first pass. (Though I'd reconsider if there turned out to be instances with many synonyms. For some as yet unspecified value of many that is...)

Looking at those 7 items:

{
  1: "Swollen abdomen (finding)",
  2: "On examination - uniform abdominal swelling (finding)",
  3: "Intra-abdominal and pelvic swelling, mass and lump (finding)",
  4: "On examination - uniform abdominal swelling (finding)",
  5: "Left lower quadrant abdominal swelling, mass, or lump (finding)",
  6: "Intra-abdominal and pelvic swelling, mass and lump (finding)",
  7: "Left lower quadrant abdominal swelling, mass, or lump (finding)"

In my entirely unqualified opinion, 7 and 5 are not synonyms - they are a more specific "child" (or whatever the Snomed language for this is) of the concept "Abdominal Swelling".

3 and 6 talk about "abdominal and pelvic", which again - to me seems like not a synonym of "Abdominal swelling", but rather a similar, but not identical concept.

2 and 4 are interesting because in the full JSON output, they have the same conceptId which I would interpret as meaning that for some reason I do not fully understand, this search returns duplicate results? In any case, I suspect that 'uniform swelling' is not the same as 'swelling' - this is something I would look to check with someone more qualified than myself though !

I don't really know what that API is for (in fact I didn't know that ihtsdotools.org had a JSON API!

However...I would imagine it to be useful to determine the main match - e.g. in this instance the concept we are looking for is 1.

My understanding is that SNOMED has a specific implementation of synonyms for concepts, and perhaps there is an API that can return them given e.g. the conceptID: 60728008 for 1 ?

Thoughts welcome...

ayamane commented 6 years ago

Hi David, No, I don't have a clinical background. I was in basic research years and years ago.

I was more interested in determining whether there was some sort of spec that outlined how the data was mapped in your example. I was initially thinking I could just write some code to populate the existing symptoms.json file, but it sounds like we would probably need to manually edit the file down to synonyms that made sense.

I can investigate the API a little more and see if I can determine whether the query can be geared a little more toward what we might need. Or maybe we just want to pull all the synonyms and see what we get?

I'll investigate the API more and report back after the weekend. -andrew

ayamane commented 6 years ago

@davidmiller I looked into the SNOMED API details some more. I found the Logical Model below in some of the documentation. Based on that since the symptom.json file contains descriptions, I would think that the best approach would be to try and pull at a minimum all Descriptions where Acceptability=Preferred. We could also grab the Acceptable Descriptions as well.

It looks as if the only path to that data though is making consecutive calls to the API. The 1st being the call to http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/descriptions?query={symptom.name}, e.g. 'Abdominal%20Swelling' Then a call to http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/concepts/{matches[0].conceptId}, e.g. '60728008' Taking the 1st match would limit the concepts to the best match of the 1st search. The resulting data from the concepts search would need to be filtered for descriptions where langMemberships has a acceptability.defaultTerm containing either Acceptable or Preferred.

I'm not positive but it appears this is how the SNOMED CT Browser would be presenting data. http://browser.ihtsdotools.org/?perspective=full&conceptId1=60728008&edition=en-edition&release=v20180131&server=http://browser.ihtsdotools.org/api/v1/snomed&langRefset=900000000000509007
So in the Abdominal Swelling example there are 2 Preferred terms and 8 Acceptable terms in the Details tab

Let me know what you think.

snomed logical model

davidmiller commented 6 years ago

I'd not seen that diagram before - it's very helpful ! :)

To work through a couple by hand as it were, what I think you're suggesting is as follows:

Altered Sensation

Takes us here: http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/descriptions?query=Altered%20Sensation where there is a direct match with code 247325003

Which gets us here: http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/concepts/247325003

That gives us 7 descriptions, looking at langMemberships.active descriptions and langMemberships.acceptability preferred/acceptable...

This would convert to the following JSON:

   {
            "name": "Altered sensation",
            "synonyms": [
    "Disturbance of skin sensation", 
    "Altered sensation of skin (finding)", 
    "Different sensation",
    "Disturbed sensation", 
    "Changed sensation", 
    "Altered sensation of skin"
],
            "coding": {
                "code": "247325003",
                "system": "http://snomed.info/sct"
// http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/concepts/247325003
             }
        }

Palpitations

Concept search: http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/descriptions?query=Palpitations

Term page: http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/concepts/80313002

This gives us the JSON

   {
            "name": "Palpitations",
            "synonyms": ["Palpitations (finding)"],
            "coding": {
                "code": "80313002",
                "system": "http://snomed.info/sct"
// http://browser.ihtsdotools.org/api/v1/snomed/en-edition/v20180131/concepts/80313002
             }
        }

Other notes

I'm aware that there's a value judgement here that Altered Sensation is the default rather than e.g. "Altered sensation of skin" or "Altered sensation of skin (finding)"... in Opal, the name value is what gets rendered on screen when something is saved as any of these...

Personally I would leave the default name and look to save screen real estate !

Honestly, I suspect that there is little actual value in having application lookuplists include both term and term (finding) - you don't want that in an autocomplete do you? I would be very tempted to exclude any 'synonyms' that were e.g. the same thing with (finding) after it!

I've discussed these sorts of things in the past with @pacharanero and @JMathiszig-Lee who may have time/inclination to weigh in :)

ayamane commented 6 years ago

I found the documentation here: https://confluence.ihtsdotools.org/display/DOCSTART/SNOMED+CT+Starter+Guide

I think what you outlined for those 2 examples is what I had in mind, although my initial thought would be to change up the order in the synonyms array to list preferred then acceptable, assuming no changes to your current model for the lookup lists.

I agree that for a lookup list the similarities between e.g. "Altered sensation of skin" and "Altered sensation of skin (finding)" are such that including both would seem excessive.

For now I can start working on filling out the sympton.json details and submit a pull request when I am finished with that.

davidmiller commented 6 years ago

change up the order in the synonyms array to list preferred then acceptable

Ah - I see !

I think in Opal the equivalents of preferred and acceptable are name and synonyms e.g. there can only be one preferred term - the name property, but many acceptable terms - the synonyms property.

There's definitely no harm in ordering the items in the synonyms array if there's a semantic meaning to the order - however worth noting that as of right now we don't actually preserve the ordering of that array when these data files are loaded into the database.

One of the reasons for that is so that you can run ./manage.py load_lookup_lists many times on the same data files without creating duplicate entries - for instance, we run it as part of deployment scripts.

So the second time the command is run, if there are new synonyms entered in the array, they will be added to the database, but the ordering of those elements isn't currently something we know about in the applications themselves.

But as well as this, we don't (yet!) have a use case for wanting to know the ordering...

The way our ForeignKeyOrFreeText fields work with the example above is more or less as follows:

If the user is completing a form with e.g. a synonyms text input, and they type disturbed sen then they get an autocomplete option for Disturbed sensation.
Once they save the form, the field's save() will figure out that this is a synonym of Altered sensation
The value that gets saved to the database is a foreign key to the Altered sensation "Preferred term"
When we look at the record in the UI, it will then render as the preferred term e.g. Altered sensation

(Similar thing happens in advanced search and query tools)

In essence synonyms are currently only really used to populate autocomplete or dropdown lists.

Hope that helps !

A PR for symptoms.json would be super !

Do shout if you come across anything you'd like a second opinion on :)

ayamane commented 6 years ago

@davidmiller I finished this up and would like to run the tests to make sure I didn't break anything. I'm not that familiar with python and couldn't find info on how/which tests to run. Do you have any quick directions?

davidmiller commented 6 years ago

Afternoon @ayamane sounds good !

We have two test suites, a python one which tests server-side code, and a javascript one that tests code that runs in the browser.

We have some docs here: https://opal.openhealthcare.org.uk/docs/guides/development_environment/ - do let me know if those don't make sense (and we can update them!)

davidmiller commented 6 years ago

The related PR for adding coding & coding system to the database is here #1571 (review pending docs) and should allow all lookuplist entries to be coded with an external terminology

JMathiszig-Lee commented 4 years ago

has this been resolved by #1622 ?

davidmiller commented 4 years ago

1622 makes it possible to store the coding - what we've not done is actually go through the list of symptoms and add codes to the entries.

List is here: https://github.com/openhealthcare/opal/blob/v0.18.4/opal/core/referencedata/data/lookuplists/symptom.json

davidmiller commented 4 years ago

In large part we haven't because we need someone with the right clinical domain knowledge to do it - if me / @fredkingham tried to go through all our terminology lists lack of e.g. medical degree means we'd misattribute n > 1. We would not know which ones we'd misattributed !

JMathiszig-Lee commented 4 years ago

Ok this is probably a good thing for me to take a crack at during a boring case

openhealthcare / opal

Snomed codes for our Symptoms lookuplist #1528

Altered Sensation

Palpitations

Other notes

1622 makes it possible to store the coding - what we've not done is actually go through the list of symptoms and add codes to the entries.