search

onc-healthit / reference-ccda-validator

Deployable C-CDA Validator war source code. Use this repository to build and deploy a validator on your local environment.
BSD 2-Clause "Simplified" License
35 stars 37 forks source link

java.util.ConcurrentModificationException in the #155

Open PeterTeunissen opened 2 years ago

PeterTeunissen commented 2 years ago

We have build the ccd-validator and are calling the rest endpoint with different documents (with different validationObjectives) concurrently under pretty high volume.

We see the below ConcurrentModificationException in the code. Looking at the code the unload/reload methods on the Mu2consolePackage.eINSTANCE look like a static operation. That makes me wonder if the ccd validator is thread safe?

java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.remove(HashMap.java:1483)
    at org.hl7.security.ds4p.contentprofile.impl.CONTENTPROFILEPackageImpl.unload(CONTENTPROFILEPackageImpl.java:435)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateDocumentByTypeUsingMDHTApi(ReferenceCCDAValidator.java:160)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateFile(ReferenceCCDAValidator.java:108)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$FastClassBySpringCGLIB$$fb042745.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$EnhancerBySpringCGLIB$$c5ebceb.validateFile(<generated>)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doMDHTValidation(ReferenceCCDAValidationService.java:202)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:127)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69)
<<shortened>>
swmuir commented 2 years ago

Hello The validations are thread safe - this particular approach of unloading validation libraries within a threaded environment was not a requirement - I will look into the issues and see about adding

Can you be more specific on how you are reproducing the issue ? Are you using the UI or the service

Thanks

Sean

On Aug 2, 2022, at 11:01 AM, Peter Teunissen @.***> wrote:

We have build the ccd-validator and are calling the code with different document which need different validationObjectives.

We see the below ConcurrentModificationException in the code. Looking at the code the unload/reload methods on the Mu2consolePackage.eINSTANCE look like a static operation. That makes me wonder if the ccd validator is thread safe?

java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.remove(HashMap.java:1483) at org.hl7.security.ds4p.contentprofile.impl.CONTENTPROFILEPackageImpl.unload(CONTENTPROFILEPackageImpl.java:435) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateDocumentByTypeUsingMDHTApi(ReferenceCCDAValidator.java:160) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateFile(ReferenceCCDAValidator.java:108) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$FastClassBySpringCGLIB$$fb042745.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$EnhancerBySpringCGLIB$$c5ebceb.validateFile() at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doMDHTValidation(ReferenceCCDAValidationService.java:202) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:127) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69) <> — Reply to this email directly, view it on GitHub https://github.com/onc-healthit/reference-ccda-validator/issues/155, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIZ4F7QQOXDEC6NFI5UVWDVXEZ6HANCNFSM55LP47GQ. You are receiving this because you are subscribed to this thread.

PeterTeunissen commented 2 years ago

It is sporadic (but persistent). I will try to recreate and update here. We have this integrated with another process, so the call is direct to the Rest endpoint (not through the UI). We will try some jmeter setup.

PeterTeunissen commented 2 years ago

There are two more flavors of these errors:

 java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.remove(HashMap.java:1483)
    at org.openhealthtools.mdht.uml.cda.mu2consol.impl.Mu2consolPackageImpl.unload(Mu2consolPackageImpl.java:303)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateDocumentByTypeUsingMDHTApi(ReferenceCCDAValidator.java:159)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateFile(ReferenceCCDAValidator.java:108)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$FastClassBySpringCGLIB$$fb042745.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
    at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$EnhancerBySpringCGLIB$$c5ebceb.validateFile(<generated>)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doMDHTValidation(ReferenceCCDAValidationService.java:202)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:127)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69)
    at com.cerner.ccdavalidator.service.CCDAValidationService.validateCCDA(CCDAValidationService.java:117)
    at com.cerner.ccdavalidator.rest.CCDAValidationController.doValidationWithDomainNameAndLogicalDomainId(CCDAValidationController.java:106)

and 

java.util.ConcurrentModificationException
    at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911)
    at java.util.ArrayList$Itr.remove(ArrayList.java:875)
    at org.sitenv.vocabularies.validation.services.VocabularyValidationService.limitConfiguredExpressionsBySeverity(VocabularyValidationService.java:174)
    at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:219)
    at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:136)
    at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:105)
    at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.doValidation(VocabularyCCDAValidator.java:78)
    at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.validateFileImplementation(VocabularyCCDAValidator.java:66)
    at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.validateFile(VocabularyCCDAValidator.java:54)
    at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator$$FastClassBySpringCGLIB$$b32d0b9.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
    at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
    at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator$$EnhancerBySpringCGLIB$$2d929969.validateFile(<generated>)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doVocabularyValidation(ReferenceCCDAValidationService.java:210)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:143)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79)
    at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69)
swmuir commented 2 years ago

Not surprised but the MU2 - as they have the same code snippet

The VocabularyValidationService is in a different area - will reach out to those resources

Sean

On Aug 2, 2022, at 3:48 PM, Peter Teunissen @.***> wrote:

There are two more flavors of these errors:

java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.remove(HashMap.java:1483) at org.openhealthtools.mdht.uml.cda.mu2consol.impl.Mu2consolPackageImpl.unload(Mu2consolPackageImpl.java:303) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateDocumentByTypeUsingMDHTApi(ReferenceCCDAValidator.java:159) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator.validateFile(ReferenceCCDAValidator.java:108) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$FastClassBySpringCGLIB$$fb042745.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.sitenv.referenceccda.validators.schema.ReferenceCCDAValidator$$EnhancerBySpringCGLIB$$c5ebceb.validateFile() at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doMDHTValidation(ReferenceCCDAValidationService.java:202) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:127) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69) at com.cerner.ccdavalidator.service.CCDAValidationService.validateCCDA(CCDAValidationService.java:117) at com.cerner.ccdavalidator.rest.CCDAValidationController.doValidationWithDomainNameAndLogicalDomainId(CCDAValidationController.java:106) and

java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911) at java.util.ArrayList$Itr.remove(ArrayList.java:875) at org.sitenv.vocabularies.validation.services.VocabularyValidationService.limitConfiguredExpressionsBySeverity(VocabularyValidationService.java:174) at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:219) at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:136) at org.sitenv.vocabularies.validation.services.VocabularyValidationService.validate(VocabularyValidationService.java:105) at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.doValidation(VocabularyCCDAValidator.java:78) at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.validateFileImplementation(VocabularyCCDAValidator.java:66) at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator.validateFile(VocabularyCCDAValidator.java:54) at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator$$FastClassBySpringCGLIB$$b32d0b9.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at org.sitenv.referenceccda.validators.vocabulary.VocabularyCCDAValidator$$EnhancerBySpringCGLIB$$2d929969.validateFile() at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.doVocabularyValidation(ReferenceCCDAValidationService.java:210) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.runValidators(ReferenceCCDAValidationService.java:143) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDAImplementation(ReferenceCCDAValidationService.java:79) at org.sitenv.referenceccda.services.ReferenceCCDAValidationService.validateCCDA(ReferenceCCDAValidationService.java:69) — Reply to this email directly, view it on GitHub https://github.com/onc-healthit/reference-ccda-validator/issues/155#issuecomment-1203147124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIZ4F2VLTMQ5GFI2Q6EEJLVXF3QZANCNFSM55LP47GQ. You are receiving this because you commented.

PeterTeunissen commented 2 years ago

We have setup jmeter, with 4 different thread groups, each with an HTTP Request with a different XML document. Each thread group makes 10 concurrent requests. So we hit the validators with 40 requests at the same time. We have 2 instances of the validator running behind a load balancer, so each instance would be hit with roughly 20 requests concurrently.

Give it enough time and we get the ConcurrentModificationException.

@swmuir We could try to help fix this, but would need some pointers. You mentioned something in the first comment about maybe not unloading validation libraries?

drbgfc commented 2 years ago

In order to work as one WAR, we had to add a load/unload procedure to switch between MU2 and non-MU2 validation. If you make a separate WAR for MU2, or, you have no use for MU2 validation and never run it (2014 is pretty old at this point), then it should be a non-issue. We could certainly look into fixing this (or removing it). We recently had a discussion where we agreed to sunset MU2 in general. So, I'm not sure we will have the resources to change the operation. It would be interesting to know if you are using MU2.