search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
208 stars 83 forks source link

Error importing 20201130 data (with Concrete Domain values) #189

Closed rbailey2 closed 1 year ago

rbailey2 commented 3 years ago

Hi,

I have set up a new Ubuntu 20.04 server, with OpenJDK version 1.8.0_275. I have downloaded and started ElasticSearch 7.7.0.

The server specs are as follows:

I'm starting ElasticSearch with "-Xms4g -Xmx4g", configured in config/jvm.options

I have downloaded Snowstorm 5.1.0

I'm attempting to import the 20201130 data using the following command:

java -Xms2g -Xmx4g -jar target/snowstorm*.jar --delete-indices --import=/home/snowstorm/NCTS_SCT_RF2_DISTRIBUTION_32506021000036107-20201130-SNAPSHOT.zip --exit

The above command fails, the following error showing up first:

2020-11-27 06:23:27.354 ERROR 56245 --- [           main] io.kaicode.elasticvc.api.BranchService   : Commit commitListener threw IllegalStateException, rolling back commit 1606456954767 on branch MAIN

java.lang.IllegalStateException: Failed to update semantic index. Failed to convert axiom EquivalentClasses(:57466011000036102 ObjectIntersectionOf(:30404011000036106 :63890011000036102 ObjectSomeValue
sFrom(:609096000 ObjectIntersectionOf(ObjectSomeValuesFrom(:177631000036102 :63011000036109) ObjectSomeValuesFrom(:30409011000036107 :54314011000036105) DataHasValue(:700000131000036101 "24"^^xsd:decim
al))) ObjectSomeValuesFrom(:700000101000036108 :4341000168101)))
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.preCommitCompletion(SemanticIndexUpdateService.java:83)
        at io.kaicode.elasticvc.api.BranchService.completeCommit(BranchService.java:398)
        at io.kaicode.elasticvc.domain.Commit.close(Commit.java:61)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.completeImportCommit(ImportComponentFactoryImpl.java:217)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.loadingComponentsCompleted(ImportComponentFactoryImpl.java:206)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:221)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:178)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.access$100(ReleaseImporter.java:164)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadSnapshotReleaseFiles(ReleaseImporter.java:44)
        at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadSnapshotReleaseFiles(ReleaseImporter.java:68)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importArchive(ImportService.java:118)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService$$FastClassBySpringCGLIB$$3479d1da.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService$$EnhancerBySpringCGLIB$$882069bb.importArchive(<generated>)
        at org.snomed.snowstorm.SnowstormApplication.importEditionRF2FromDisk(SnowstormApplication.java:135)
        at org.snomed.snowstorm.SnowstormApplication.run(SnowstormApplication.java:95)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:786)
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
        at org.snomed.snowstorm.SnowstormApplication.main(SnowstormApplication.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.snomed.otf.owltoolkit.conversion.ConversionException: Failed to convert axiom EquivalentClasses(:57466011000036102 ObjectIntersectionOf(:30404011000036106 :63890011000036102 ObjectSomeValuesFrom(:609096000 ObjectIntersectionOf(ObjectSomeValuesFrom(:177631000036102 :63011000036109) ObjectSomeValuesFrom(:30409011000036107 :54314011000036105) DataHasValue(:700000131000036101 "24"^^xsd:decimal))) ObjectSomeValuesFrom(:700000101000036108 :4341000168101)))
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.lambda$axiomStreamToRelationshipStream$20(SemanticIndexUpdateService.java:578)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.axiomStreamToRelationshipStream(SemanticIndexUpdateService.java:558)
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.buildRelevantPartsOfExistingGraph(SemanticIndexUpdateService.java:376)
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.updateSemanticIndex(SemanticIndexUpdateService.java:141)
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.updateStatedAndInferredSemanticIndex(SemanticIndexUpdateService.java:108)
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.preCommitCompletion(SemanticIndexUpdateService.java:81)
        ... 30 common frames omitted
Caused by: org.snomed.otf.owltoolkit.conversion.ConversionException: Expecting ObjectSomeValuesFrom within ObjectIntersectionOf as part of role group, got DataHasValue in expression ObjectIntersectionOf(<http://snomed.info/id/30404011000036106> <http://snomed.info/id/63890011000036102> ObjectSomeValuesFrom(<http://snomed.info/id/609096000> ObjectIntersectionOf(ObjectSomeValuesFrom(<http://snomed.info/id/177631000036102> <http://snomed.info/id/63011000036109>) ObjectSomeValuesFrom(<http://snomed.info/id/30409011000036107> <http://snomed.info/id/54314011000036105>) DataHasValue(<http://snomed.info/id/700000131000036101> "24"^^xsd:decimal))) ObjectSomeValuesFrom(<http://snomed.info/id/700000101000036108> <http://snomed.info/id/4341000168101>)).
        at org.snomed.otf.owltoolkit.conversion.AxiomRelationshipConversionService.getRelationships(AxiomRelationshipConversionService.java:261)
        at org.snomed.otf.owltoolkit.conversion.AxiomRelationshipConversionService.convertAxiomToRelationships(AxiomRelationshipConversionService.java:120)
        at org.snomed.otf.owltoolkit.conversion.AxiomRelationshipConversionService.convertAxiomToRelationships(AxiomRelationshipConversionService.java:57)
        at org.snomed.snowstorm.core.data.services.AxiomConversionService.convertAxiomMemberToAxiomRepresentation(AxiomConversionService.java:42)
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.lambda$axiomStreamToRelationshipStream$20(SemanticIndexUpdateService.java:560)
        ... 36 common frames omitted

The same occurs when I attempt to use --import-full and the full data zip file.

Am I doing something wrong? I've attached the Snowstorm and ElasticSearch logs.

elasticsearch-2020-11-27-1.log.gz snowstorm-5.1.0_output.log

Thanks, Rob

CoderMChu commented 3 years ago

Hi Rob,

It looks like you have concrete domain modelling (DataHasValue(:700000131000036101 "24"^^xsd:decimal) in the snapshot file and snowstorm 5.1.0 isn't supporting this yet. You need the develop version of Snowstorm for axioms with concrete domain.

Thanks

Michael

kaicode commented 3 years ago

Hi @rbailey2, Further to this.. the development work for supporting for concrete domains has only just started. We have done some work on the RF2 import but concrete domain support for viewing concepts, ECL, RF2 export, classification and many other things have not been developed yet.

We will release a version of Snowstorm with support for viewing content with concrete domains by the end of January in time for the Concrete Domains International Edition Technical Preview around the end of Jan.

We will release a version of Snowstorm with support for authoring content with concrete domains during the first half of 2021.

Kind regards, Kai

rbailey2 commented 3 years ago

Hi Michael and Kai,

Thanks for the response. I'll check back after January 👍

Cheers, Rob

olamide226 commented 3 years ago

I have a similar problem with this as well. However, I am using the latest snowstorm release jar.

java -Xms2g -Xmx4g -jar snowstorm-6.0.3.jar --delete-indices --import=~/Downloads/SnomedCT_Release_AU1000036_20210131.zip --exit

See error.



java.lang.IllegalStateException: Failed to update semantic index. Failed to convert axiom EquivalentClasses(:782706003 ObjectSomeValuesFrom(:609096000 ObjectSomeValuesFrom(:424876005 :260635009)))
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.preCommitCompletion(SemanticIndexUpdateService.java:92)
    at io.kaicode.elasticvc.api.BranchService.completeCommit(BranchService.java:398)
    at io.kaicode.elasticvc.domain.Commit.close(Commit.java:61)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.completeImportCommit(ImportComponentFactoryImpl.java:217)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportComponentFactoryImpl.loadingComponentsCompleted(ImportComponentFactoryImpl.java:206)
    at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:228)
    at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.doLoadReleaseFiles(ReleaseImporter.java:180)
    at org.ihtsdo.otf.snomedboot.ReleaseImporter$ImportRun.access$100(ReleaseImporter.java:164)
    at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadSnapshotReleaseFiles(ReleaseImporter.java:44)
    at org.ihtsdo.otf.snomedboot.ReleaseImporter.loadSnapshotReleaseFiles(ReleaseImporter.java:68)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportService.snapshotImport(ImportService.java:209)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importFiles(ImportService.java:167)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importArchive(ImportService.java:118)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportService$$FastClassBySpringCGLIB$$3479d1da.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
    at org.snomed.snowstorm.core.rf2.rf2import.ImportService$$EnhancerBySpringCGLIB$$5e889626.importArchive(<generated>)
    at org.snomed.snowstorm.SnowstormApplication.importEditionRF2FromDisk(SnowstormApplication.java:135)
    at org.snomed.snowstorm.SnowstormApplication.run(SnowstormApplication.java:95)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:786)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at org.snomed.snowstorm.SnowstormApplication.main(SnowstormApplication.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.snomed.otf.owltoolkit.conversion.ConversionException: Failed to convert axiom EquivalentClasses(:782706003 ObjectSomeValuesFrom(:609096000 ObjectSomeValuesFrom(:424876005 :260635009)))
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.lambda$axiomStreamToRelationshipStream$23(SemanticIndexUpdateService.java:630)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.axiomStreamToRelationshipStream(SemanticIndexUpdateService.java:610)
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.buildRelevantPartsOfExistingGraph(SemanticIndexUpdateService.java:428)
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.updateSemanticIndex(SemanticIndexUpdateService.java:150)
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.updateStatedAndInferredSemanticIndex(SemanticIndexUpdateService.java:117)
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.preCommitCompletion(SemanticIndexUpdateService.java:90)
    ... 32 common frames omitted
Caused by: org.snomed.otf.owltoolkit.conversion.ConversionException: Expecting ObjectIntersectionOf at first level of expression, got ObjectSomeValuesFrom in expression ObjectSomeValuesFrom(<http://snomed.info/id/609096000> ObjectSomeValuesFrom(<http://snomed.info/id/424876005> <http://snomed.info/id/260635009>)).
    at org.snomed.otf.owltoolkit.conversion.AxiomRelationshipConversionService.getRelationships(AxiomRelationshipConversionService.java:282)
    at org.snomed.otf.owltoolkit.conversion.AxiomRelationshipConversionService.convertAxiomToRelationships(AxiomRelationshipConversionService.java:156)
    at org.snomed.otf.owltoolkit.conversion.AxiomRelationshipConversionService.convertAxiomToRelationships(AxiomRelationshipConversionService.java:56)
    at org.snomed.snowstorm.core.data.services.AxiomConversionService.convertAxiomMemberToAxiomRepresentation(AxiomConversionService.java:43)
    at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.lambda$axiomStreamToRelationshipStream$23(SemanticIndexUpdateService.java:612)
    ... 38 common frames omitted ```
kaicode commented 3 years ago

Hi @rbailey2,

The axiom you are trying to load seems to be malformed EquivalentClasses(:782706003 ObjectSomeValuesFrom(:609096000 ObjectSomeValuesFrom(:424876005 :260635009))).

A formatted version of this axiom looks like this:

EquivalentClasses(
    :782706003 | Subconjunctival insertion of transcleral gelatin stent (procedure) |
    ObjectSomeValuesFrom(
        :609096000 | Role group (attribute) |
        ObjectSomeValuesFrom(
            :424876005 | Surgical approach (attribute) |
            :260635009 | Transcleral approach (qualifier value) |
        )
    )
)

This appears to be the axiom of a defined concept but the format of the axiom is unexpected because no parent concept has been stated. Our tooling expects axioms which have attributes to include ObjectIntersectionOf, containing the parents and the attributes. For example:

EquivalentClasses(
    :782706003 | Subconjunctival insertion of transcleral gelatin stent (procedure) |
    ObjectIntersectionOf(
        :71388002 |Procedure (procedure)|
        ObjectSomeValuesFrom(
            :609096000 | Role group (attribute) |
            ObjectSomeValuesFrom(
                :424876005 | Surgical approach (attribute) |
                :260635009 | Transcleral approach (qualifier value) |
            )
        )
    )
)

The complete axiom for this concept from the International Edition can be seen in the SNOMED CT Browser in the "Expression" tab.

olamide226 commented 3 years ago

Thanks for this @kaicode. Is there a way around loading other editions with axioms without parent concepts? Or the system is limited to loading Int'l edition of RF2 files for now.

kaicode commented 3 years ago

We have successfully loaded 13 editions and extensions, other than the International Edition, into Snowstorm (see SNOMED CT Browser which runs off the Snowstorm API).

This axiom is very unusual. My understanding is that the AU release does not usually contain axioms, only inferred relationships, it is a special package that you have?

olamide226 commented 3 years ago

Not a special package. I got the most recent release from the Australian Digital Health centre. However I am able to load the Int'l edition into snowstorm. So it must be the AU release with the issue.

kaicode commented 3 years ago

@olamide226 I've downloaded the AU release NCTS_SCT_RF2_DISTRIBUTION_32506021000036107-20210228-ALL.zip to take a look. There are two axioms for the concept 782706003, one is active and one is inactive. It's the inactive axiom that has the unexpected format, this row:

dd71a742-e976-47c0-b9e6-ae3cf4271563    20200331    0   32506021000036107   733073007   782706003   EquivalentClasses(:782706003 ObjectSomeValuesFrom(:609096000 ObjectSomeValuesFrom(:424876005 :260635009)))

An easy workaround to get the AU extension to load would be to remove this row from the SnomedCT_Release_AU1000036_20210228/RF2Release/Snapshot/Terminology/sct2_sRefset_OWLExpressionSnapshot_AU1000036_20210228.txt file within the zip file before loading.

kaicode commented 1 year ago

Closing this issue due to inactivity.