search

timsbiomed / owl-on-fhir-content

Static OWL content, pre and post converted to FHIR.
MIT License
0 stars 0 forks source link

referenced, but not defined codes #5

Open chrisroederucdenver opened 1 year ago

chrisroederucdenver commented 1 year ago

After fixing the other two bugs in the json file, the data still wouldn't load. I'm not there yet, but the process runs longer after removing references to concepts that don't exist. The references look like this:

"valueCode": "https://loinc.org/LP33103-0"
"valueCode": "https://loinc.org/LP33117-0"
"valueCode": "https://loinc.org/lc0000001"

back in the merged_reasoned_loinc.owl , the concepts exist in a single line each, that I'm guessing are the definitions.

    <owl:Class rdf:about="https://loinc.org/lc0000001">
   <owl:Class rdf:about="https://loinc.org/LP33117-0">
    <owl:Class rdf:about="https://loinc.org/LP33103-0">

More detail about these guys:


    <owl:Class rdf:about="https://loinc.org/lc0000001">
        <rdfs:subClassOf rdf:resource="http://www.w3.org/2002/07/owl#Thing"/>
    </owl:Class>

    <owl:Class rdf:about="https://loinc.org/LP33117-0">
        <rdfs:subClassOf rdf:resource="http://www.w3.org/2002/07/owl#Thing"/>
    </owl:Class>

   <owl:Class rdf:about="https://loinc.org/LP33103-0">
        <rdfs:subClassOf rdf:resource="http://www.w3.org/2002/07/owl#Thing"/>
    </owl:Class>

## Related
- https://github.com/geneontology/obographs/issues/90
- https://github.com/ontodev/robot/issues/1082
joeflack4 commented 1 year ago

Hey @chrisroederucdenver Sorry to see you working late on the weekend on this. I thought @ShahimEssaid was going to load these earlier.

It looks like some of them just don't have labels. I'm not sure why..

Just a couple notes of some things that look to be contradictory. LP33032-1 is in your last snippet, but not in the first two. LP33117-0 is in the first two, but not in the last. So maybe you meant one or the other of these.

chrisroederucdenver commented 1 year ago

LP33117-0 owl example fixed.

How does a missing label impact those concepts? Can you show examples with and without to be more clear?

chrisroederucdenver commented 1 year ago

As far as testing these files, I've been maintaining jpaserver-scripts to set up a server and load data. It's what I've been using and what I used this evening. You should be able to run them yourself on your mac and test your work.

joeflack4 commented 1 year ago

Oh, sorry. I don't see why a missing label would cause an issue, but when you said they were undefined I wasn't sure what you meant. The last snippet you had looked like appropriate class definitions to me.

That sounds good to me, the jpaserver-scripts. I thought Shahim was going to load them / let me know if there was an issue at this point. My plan was just to curl POST them to our live server, at least before the connectathon.

chrisroederucdenver commented 1 year ago

Waiting until the last minute to find potential bugs rarely turns out well. Even if there weren't any bugs, knowing this, from a successful load, is tremendously valuable and relieving. Instead of worrying about having data, I could be worrying about the other things on my plate.

Please test your code, and do it early, do it thoroughly and do it often.

Get Outlook for iOShttps://aka.ms/o0ukef

From: Joe Flack @.> Sent: Saturday, January 7, 2023 10:10:26 PM To: HOT-Ecosystem/owl-on-fhir-content @.> Cc: Roeder, Chris3 @.>; Mention @.> Subject: Re: [HOT-Ecosystem/owl-on-fhir-content] referenced, but not defined codes (Issue #5)

[External Email - Use Caution]

Oh, sorry. I don't see why a missing label would cause an issue, but when you said they were undefined I wasn't sure what you meant. The last snippet you had looked like appropriate class definitions to me.

That sounds good to me, the jpaserver-scripts. I thought Shahim was going to load them / let me know if there was an issue at this point. My plan was just to curl POST them to our live server, at least before the connectathon.

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FHOT-Ecosystem%2Fowl-on-fhir-content%2Fissues%2F5%23issuecomment-1374711205&data=05%7C01%7Cchris.roeder%40cuanschutz.edu%7C9bc56ba959894d564ac608daf136a78b%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638087514294615842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TFHWUdFWdPKq28Qw5K0rS6r7jRWObKkfcw7G15VJWzw%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALG3NFJMYG2XJ5EVNFL7TADWRJD4FANCNFSM6AAAAAATUKS3UY&data=05%7C01%7Cchris.roeder%40cuanschutz.edu%7C9bc56ba959894d564ac608daf136a78b%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638087514294615842%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SXSvVZy2TFUBZVHaj2agvHR%2BPMOMJfRRbbPcUEhifTs%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

joeflack4 commented 1 year ago

Hey Chris. I don't disagree w/ that. I'm really sorry about this. I do have unit tests. They're just not for the server upload. I haven't had a lot of time. I was trying to upload via the UI, but when I told you guys about this, Shahim said he would upload for me. I agree with you this is not good at this stage.

joeflack4 commented 1 year ago

Also, btw now that I think about it, it could be because the work is not actually finished yet. It's been a lot slower since OAK. But there are still some easy things to do. I think it's just missing some required top level properties from CodeSystem, like status and content.

chrisroederucdenver commented 1 year ago

Thanks.

I don’t want to back you into a corner insisting on perfection. We moving fast on the bleeding edge. XXit happens.

I also know/assume you have to share the file with Shahim somehow, and the git repo does that job. One way to use it, yet counter the common assumption that something committed has been tested is to include that in the commit message or branch name like “first_fhir_untested”

Also, I’ve put a non-trivial amount off effort into jpaserver-scripts. I initially wrote them for myself, but efforts to improve the README etc. are done to empower others to set up their own servers, and be able to test with. I’ve since gotten to know the options in application.yaml a little better. It runs pretty fast when you don’t include an IG. We need them, but not in all cases…

chrisroederucdenver commented 1 year ago

OAK is slower. I've used breaks like that to work on other things, like the TimsUI. When its crunch time, you have to be creative.

TRIM YOUR TEST INPUT DOWN TO SIZE.

When you get the small, like 1/100th size, one working, build a larger file and find out what bugs were lurking there. rinse. repeat.

If you haven't, have a look at the unix tools head and tail. I did this sort of thing to narrow down the json when looking for that bug with the missing references. ` unix> head -1000 file.owl > file_short.owl Then go into Vi (or other editor) to clean up the last map and list with } and ].

joeflack4 commented 1 year ago

Thanks Chris. This is all stuff I'm familiar with. By OAK being slower, I don't mean the conversion process, I mean the development process.

joeflack4 commented 1 year ago

I looked and this seems to be an Obographs issue: https://github.com/ontodev/robot/issues/1082 / https://github.com/geneontology/obographs/issues/90

I'll put in a workaround for this for now.