search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
204 stars 80 forks source link

Import Extensions not working(SNOMED CT India Extensions) #496

Open sudheer-unify-care opened 1 year ago

sudheer-unify-care commented 1 year ago

Snowstorm Version: 8.1.0

Below are the errors that I'm getting while uploading SNOMED CT India extensions: [POST] {{opsUrl}}/imports/1a454b3b-fe12-4013-89e3-4d0b8692c5ef/archive

Geographical Locations Extensions Error

Failed RF2 SNAPSHOT import on branch MAIN/SNOMEDCT-INGLE. ID 1a454b3b-fe12-4013-89e3-4d0b8692c5ef

java.lang.NullPointerException: null
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.findAncestorModules(ModuleDependencyService.java:375)
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.determineClosestParentFromExistingHierarchy(ModuleDependencyService.java:351)
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.mapTopLevelExtensionModuleToCore(ModuleDependencyService.java:299)
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.generateModuleDependencies(ModuleDependencyService.java:195)
        at org.snomed.snowstorm.core.data.services.ReleaseService.createVersion(ReleaseService.java:55)
        at org.snomed.snowstorm.core.data.services.CodeSystemService.createVersion(CodeSystemService.java:239)
        at org.snomed.snowstorm.core.data.services.CodeSystemService.createVersionIfCodeSystemFoundOnPath(CodeSystemService.java:318)
        at org.snomed.snowstorm.core.data.services.CodeSystemService$$FastClassBySpringCGLIB$$7d61f252.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386)
        at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704)
        at org.snomed.snowstorm.core.data.services.CodeSystemService$$EnhancerBySpringCGLIB$$ce53c6c0.createVersionIfCodeSystemFoundOnPath(<generated>)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importArchive(ImportService.java:115)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.lambda$importArchiveAsync$1(ImportService.java:261)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Patient Instructions Extensions Error

Failed RF2 SNAPSHOT import on branch MAIN/SNOMEDCT-INPIE. ID cb55bdd9-171f-4954-ad83-8367cd8cbfdf

java.lang.NullPointerException: null
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.findAncestorModules(ModuleDependencyService.java:375)
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.determineClosestParentFromExistingHierarchy(ModuleDependencyService.java:351)
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.mapTopLevelExtensionModuleToCore(ModuleDependencyService.java:299)
        at org.snomed.snowstorm.core.data.services.ModuleDependencyService.generateModuleDependencies(ModuleDependencyService.java:195)
        at org.snomed.snowstorm.core.data.services.ReleaseService.createVersion(ReleaseService.java:55)
        at org.snomed.snowstorm.core.data.services.CodeSystemService.createVersion(CodeSystemService.java:239)
        at org.snomed.snowstorm.core.data.services.CodeSystemService.createVersionIfCodeSystemFoundOnPath(CodeSystemService.java:318)
        at org.snomed.snowstorm.core.data.services.CodeSystemService$$FastClassBySpringCGLIB$$7d61f252.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386)
        at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704)
        at org.snomed.snowstorm.core.data.services.CodeSystemService$$EnhancerBySpringCGLIB$$ce53c6c0.createVersionIfCodeSystemFoundOnPath(<generated>)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.importArchive(ImportService.java:115)
        at org.snomed.snowstorm.core.rf2.rf2import.ImportService.lambda$importArchiveAsync$1(ImportService.java:261)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Above 2 errors are same, added for reference

Reference Set Extension: Error

Failed RF2 SNAPSHOT import on branch MAIN/SNOMEDCT-INRSE. ID 2ee16de7-c1af-4f14-8a96-9e44ee2a3d37

org.ihtsdo.otf.snomedboot.ReleaseImportException: Failed to load release files during release import process. 1 exceptions caught in threads. First exception: Invalid RF2 content. 'id' not found as first value in tab separated first line of der2_Refset_cardiothoracicAndVascularSurgerySnapshot_IN1000189_20210806.txt.

COVID 19 Extension: Error

2023-03-25 13:32:55.056 ERROR 33741 --- [pool-2-thread-1] o.s.s.core.rf2.rf2import.ImportService   : Failed RF2 SNAPSHOT import on branch MAIN/SNOMEDCT-INCV. ID 59edff45-9736-4554-9c98-7b760e3bae16

java.lang.IllegalStateException: Failed to update semantic index. Loop found in transitive closure for concept 49062001 on branch MAIN/SNOMEDCT-INCV. The concept 49062001 is in its own set of ancestors: [760191000189109, 49062001, 260787004, 138875005]
        at org.snomed.snowstorm.core.data.services.SemanticIndexUpdateService.preCommitCompletion

I'm able to upload two other extensions successfully(Drug Codes & AYUSH). Please let me know , if you need any other information.

kaicode commented 1 year ago

Unfortunately of these errors point to quality issues within the package being loaded.

The behaviour of the ModuleDependencyService should be improved but it throws this error when there are missing rows in the module dependency refset within the package being loaded. It expects there to be a module dependency row for every module used in the package. I would be happy to suggest the missing rows if you give me access to the package.

The Reference Set Extension error looks like it is caused by non-standard column naming. The SNOMED CT RF2 specification states that the first six columns of all reference sets must be named id, effectiveTime, active, moduleId, refsetId and referencedComponentId. There may be additional columns after these.

The COVID 19 Extension error is caused by a circular loop between concepts when following the is-a relationships. Snowstorm traces each concept's parents up to the root concept using both the stated view (axioms) and the inferred view (inferred relationships). The rest of that error message should say if it was the semantic index for the stated form or the inferred form that caused this "transitive closure loop".
It looks like concept 760191000189109 is a 49062001 |Device (physical object)| but perhaps the 49062001 |Device (physical object)| concept has been given a parent concept of 760191000189109?

These are all content issues within the RF2 packages that should be fixed by the SNOMED CT extension maintainer. Unfortunately the packages can not be loaded into Snowstorm until these issues are fixed.

sudheer-unify-care commented 1 year ago

@kaicode If I reach SNOMED CT extension maintainers, not sure how much time they will to fix this. Excluding Reference Set error, is there a way to gracefully upload by ignore the wrong codes/relationships?

I've all extensions zip files in my local, How can I provide the access?

kaicode commented 1 year ago

The CDAC website seems to be down at the moment. Please reach out to the extension maintainers to give them a chance to respond first. I am at the SNOMED CT business conference next week but should have some time to suggest workarounds the following week.

sudheer-unify-care commented 1 year ago

Yes, I already notified maintainers and shared the issue details as well.

Just wanted to check with you for any work around to unblock myself.

Anyways, thanks for quick turn around. Will wait for the response.

kaicode commented 1 year ago

There is no config or quick fix to work around this at the moment. Fixing the content one way or another is the only way to get the import to complete.

On Wed, 29 Mar 2023, 18:15 sudheer-unify-care, @.***> wrote:

Yes, I already notified maintainers and shared the issue details as well.

Just wanted to check with you for any work around to unblock myself.

Anyways, thanks for quick turn around. Will wait for the response.

— Reply to this email directly, view it on GitHub https://github.com/IHTSDO/snowstorm/issues/496#issuecomment-1488998422, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPK3FMVUUSLVJICEVW4X7LW6RU4TANCNFSM6AAAAAAWHN4AXI . You are receiving this because you were mentioned.Message ID: @.***>

-- --- SNOMED International is the trading name of the International Health Terminology Standards Development Organisation (IHTSDO), a private company limited by guarantee, registered in England and Wales, Company Registration Number 9915820, reg. address One Kingdom Street, Paddington Central, London W2 6BD, UK.

This communication, its contents and any files attached are confidential and may contain privileged information. If you are not the intended recipient please notify the sender immediately by reply e-mail and please delete the message from your system immediately. You should not copy it or use it for any purpose nor disclose its contents to any other person.