search

ppazos / cabolabs-ehrserver

Open platform to manage and share standardized clinical data, designed by @ppazos at CaboLabs Health Informatics.
https://cabolabs.com
Apache License 2.0
181 stars 110 forks source link

Check why some compositions are not being indexed in staging #344

Closed ppazos closed 8 years ago

ppazos commented 8 years ago

2016-05-26 04:18:05,097 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/55e41957-0b26-415f-9dcd-10d047d12827_EMR_1.xml 2016-05-26 04:18:05,099 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/c4442184-3ef8-4eca-97b4-1a8f82fd2bc3_EMR_1.xml 2016-05-26 04:18:05,100 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6_EMR_1.xml 2016-05-26 04:18:05,101 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/92ed0b4a-8872-42a3-a558-949694c4b887_EMR_1.xml 2016-05-26 04:18:05,102 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/2da3fefb-b74f-4596-9e68-89b1f3c33d63_EMR_1.xml 2016-05-26 04:18:05,103 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/8814c6fa-6b1e-407c-8e8f-f6e0214a226a_EMR_1.xml 2016-05-26 04:18:05,104 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/85d0473b-8331-4455-802f-43b5dbbe4345_EMR_1.xml 2016-05-26 04:18:05,105 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/f181d1ff-868c-47fc-a6b9-1aa747f61f68_EMR_1.xml 2016-05-26 04:18:05,107 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/b3e9c632-9ff8-450b-93cf-eb384447e972_EMR_1.xml 2016-05-26 04:18:05,108 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/c7b17c2c-052c-410a-aea1-4b5868c4d55c_EMR_1.xml 2016-05-26 04:18:05,109 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/61a1fcfa-ae03-450e-af12-36df6ec1e7db_EMR_1.xml 2016-05-26 04:18:05,110 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/1c6fa300-022f-4d2b-849d-956d64cd806c_EMR_1.xml 2016-05-26 04:18:50,098 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/55e41957-0b26-415f-9dcd-10d047d12827_EMR_1.xml 2016-05-26 04:18:50,098 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/c4442184-3ef8-4eca-97b4-1a8f82fd2bc3_EMR_1.xml 2016-05-26 04:18:50,099 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6_EMR_1.xml 2016-05-26 04:18:50,100 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/92ed0b4a-8872-42a3-a558-949694c4b887_EMR_1.xml 2016-05-26 04:18:50,101 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/2da3fefb-b74f-4596-9e68-89b1f3c33d63_EMR_1.xml 2016-05-26 04:18:50,102 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/8814c6fa-6b1e-407c-8e8f-f6e0214a226a_EMR_1.xml 2016-05-26 04:18:50,103 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/85d0473b-8331-4455-802f-43b5dbbe4345_EMR_1.xml 2016-05-26 04:18:50,104 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/f181d1ff-868c-47fc-a6b9-1aa747f61f68_EMR_1.xml 2016-05-26 04:18:50,105 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/b3e9c632-9ff8-450b-93cf-eb384447e972_EMR_1.xml 2016-05-26 04:18:50,106 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/c7b17c2c-052c-410a-aea1-4b5868c4d55c_EMR_1.xml 2016-05-26 04:18:50,107 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/61a1fcfa-ae03-450e-af12-36df6ec1e7db_EMR_1.xml 2016-05-26 04:18:50,108 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing /var/lib/openshift/555167ba5973ca0a3d000113/jb ossews/versions/1c6fa300-022f-4d2b-849d-956d64cd806c_EMR_1.xml

ppazos commented 8 years ago

It seems these are problems generated by failed rollbacks on commits. For some reason the XmlService couldn't write the version files to disk (https://github.com/ppazos/cabolabs-ehrserver/blob/master/grails-app/services/com/cabolabs/ehrserver/parsers/XmlService.groovy#L224-L251) and when the indexing job fails because the file is not on disk but the compo index data is on the database (https://github.com/ppazos/cabolabs-ehrserver/blob/master/grails-app/services/com/cabolabs/ehrserver/data/DataIndexerService.groovy#L50-L57)

This might be solved with https://github.com/ppazos/cabolabs-ehrserver/issues/282

We still need to clean the database because of the orphan records on composition index and version, and dependencies (contribution, etc)

ppazos commented 8 years ago

We are throwing RuntimeExceptions from the XmlService to rollback the transactions, it seems this change was made after the dates the failing versions were committed.

The problem is we don't have the original file committed by the client app, just records on the database, because the commit process failed to save the files.

For that reason we need to make a raw log of all the committed XML, just in case something goes wrong with the commit and the client app doesn't commit the file again. Of course that would be responsibility of the client apps, since they receive all the commit errors. The problem is that the server leaves data in the DB but the files are not on the version repo. But this case shouldn't happen because now the rollback is working OK and the integration tests are testing all the commit fail cases and assuring we don't have records in the database and files on the filesystem that shouldn't be there after a failed commit.

ppazos commented 8 years ago

example version 811 with uid e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6::EMR::1 references compositionIndex 811 and doesnt have a matching xml in /versions e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6_EMR_1.xml

the same for the others, I need to update the DB manually to clean this and store each commit as a secure log, even if the commit fails, to log the fails also.

ppazos commented 8 years ago

Due latest refactor, now instead of skipping an exception is thrown:

2016-07-19 00:50:47,180 [quartzScheduler_Worker-3] ERROR listeners.ExceptionPrinterJobListener - Exception occurred in job: Grails Job org.quartz.JobExecutionException: java.lang.reflect.UndeclaredThrowableException [See nested exception: java.lang.reflect.UndeclaredThrowableException] at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:111) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) Caused by: java.lang.reflect.UndeclaredThrowableException at com.cabolabs.ehrserver.data.DataIndexerService$ttgenerateIndexes_closure3.doCall(DataIndexerService.groovy:55) at com.cabolabs.ehrserver.data.DataIndexerService.$tt__generateIndexes(DataIndexerService.groovy:44) at com.cabolabs.ehrserver.indexing.IndexDataJob.execute(IndexDataJob.groovy:23) at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104) ... 2 more Caused by: java.io.FileNotFoundException: File versions/55e41957-0b26-415f-9dcd-10d047d12827_EMR_1.xml doesn't exists at com.cabolabs.ehrserver.versions.VersionFSRepoService.getExistingVersionFile(VersionFSRepoService.groovy:54) ... 6 more

Check the exception and catch it.

ppazos commented 8 years ago

The problems is related to this: http://stackoverflow.com/questions/19987720/exception-thrown-from-service-not-being-caught-in-controller

The exception is thrown correctly but the catch expects a different type of except, groovy is wrapping our except in an undeclaredthrowableexception.

ppazos commented 8 years ago

The problems about undeclaredthrowableexception was fixed.

ppazos commented 8 years ago

Check if there are new ones:

2016-08-13 13:47:05,995 [quartzScheduler_Worker-1] INFO data.DataIndexerService - avoid indexing version de248ecb-94cd-43a7-b5ab-0d6a4d43bc64::EMR::1 File versions/de248ecb-94cd-43a7-b5ab-0d6a4d43bc64_EMR_1.xml doesn't exists 2016-08-13 13:47:39,808 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 55e41957-0b26-415f-9dcd-10d047d12827::EMR::1 File versions/55e41957-0b26-415f-9dcd-10d047d12827_EMR_1.xml doesn't exists 2016-08-13 13:47:39,809 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version c4442184-3ef8-4eca-97b4-1a8f82fd2bc3::EMR::1 File versions/c4442184-3ef8-4eca-97b4-1a8f82fd2bc3_EMR_1.xml doesn't exists 2016-08-13 13:47:39,811 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6::EMR::1 File versions/e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6_EMR_1.xml doesn't exists 2016-08-13 13:47:39,812 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 92ed0b4a-8872-42a3-a558-949694c4b887::EMR::1 File versions/92ed0b4a-8872-42a3-a558-949694c4b887_EMR_1.xml doesn't exists 2016-08-13 13:47:39,813 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 2da3fefb-b74f-4596-9e68-89b1f3c33d63::EMR::1 File versions/2da3fefb-b74f-4596-9e68-89b1f3c33d63_EMR_1.xml doesn't exists 2016-08-13 13:47:39,815 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 8814c6fa-6b1e-407c-8e8f-f6e0214a226a::EMR::1 File versions/8814c6fa-6b1e-407c-8e8f-f6e0214a226a_EMR_1.xml doesn't exists 2016-08-13 13:47:39,816 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 85d0473b-8331-4455-802f-43b5dbbe4345::EMR::1 File versions/85d0473b-8331-4455-802f-43b5dbbe4345_EMR_1.xml doesn't exists 2016-08-13 13:47:39,817 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version f181d1ff-868c-47fc-a6b9-1aa747f61f68::EMR::1 File versions/f181d1ff-868c-47fc-a6b9-1aa747f61f68_EMR_1.xml doesn't exists 2016-08-13 13:47:39,818 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version b3e9c632-9ff8-450b-93cf-eb384447e972::EMR::1 File versions/b3e9c632-9ff8-450b-93cf-eb384447e972_EMR_1.xml doesn't exists 2016-08-13 13:47:39,820 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version c7b17c2c-052c-410a-aea1-4b5868c4d55c::EMR::1 File versions/c7b17c2c-052c-410a-aea1-4b5868c4d55c_EMR_1.xml doesn't exists 2016-08-13 13:47:39,821 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 61a1fcfa-ae03-450e-af12-36df6ec1e7db::EMR::1 File versions/61a1fcfa-ae03-450e-af12-36df6ec1e7db_EMR_1.xml doesn't exists 2016-08-13 13:47:39,877 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version 1c6fa300-022f-4d2b-849d-956d64cd806c::EMR::1 File versions/1c6fa300-022f-4d2b-849d-956d64cd806c_EMR_1.xml doesn't exists 2016-08-13 13:47:39,881 [quartzScheduler_Worker-2] INFO data.DataIndexerService - avoid indexing version de248ecb-94cd-43a7-b5ab-0d6a4d43bc64::EMR::1 File versions/de248ecb-94cd-43a7-b5ab-0d6a4d43bc64_EMR_1.xml doesn't exists 2016-08-13 13:48:24,807 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 55e41957-0b26-415f-9dcd-10d047d12827::EMR::1 File versions/55e41957-0b26-415f-9dcd-10d047d12827_EMR_1.xml doesn't exists 2016-08-13 13:48:24,808 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version c4442184-3ef8-4eca-97b4-1a8f82fd2bc3::EMR::1 File versions/c4442184-3ef8-4eca-97b4-1a8f82fd2bc3_EMR_1.xml doesn't exists 2016-08-13 13:48:24,810 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6::EMR::1 File versions/e8eb2086-78b2-4f5f-ad2a-41bb4eaf1cb6_EMR_1.xml doesn't exists 2016-08-13 13:48:24,811 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 92ed0b4a-8872-42a3-a558-949694c4b887::EMR::1 File versions/92ed0b4a-8872-42a3-a558-949694c4b887_EMR_1.xml doesn't exists 2016-08-13 13:48:24,812 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 2da3fefb-b74f-4596-9e68-89b1f3c33d63::EMR::1 File versions/2da3fefb-b74f-4596-9e68-89b1f3c33d63_EMR_1.xml doesn't exists 2016-08-13 13:48:24,814 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 8814c6fa-6b1e-407c-8e8f-f6e0214a226a::EMR::1 File versions/8814c6fa-6b1e-407c-8e8f-f6e0214a226a_EMR_1.xml doesn't exists 2016-08-13 13:48:24,815 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 85d0473b-8331-4455-802f-43b5dbbe4345::EMR::1 File versions/85d0473b-8331-4455-802f-43b5dbbe4345_EMR_1.xml doesn't exists 2016-08-13 13:48:24,816 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version f181d1ff-868c-47fc-a6b9-1aa747f61f68::EMR::1 File versions/f181d1ff-868c-47fc-a6b9-1aa747f61f68_EMR_1.xml doesn't exists 2016-08-13 13:48:24,818 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version b3e9c632-9ff8-450b-93cf-eb384447e972::EMR::1 File versions/b3e9c632-9ff8-450b-93cf-eb384447e972_EMR_1.xml doesn't exists 2016-08-13 13:48:24,819 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version c7b17c2c-052c-410a-aea1-4b5868c4d55c::EMR::1 File versions/c7b17c2c-052c-410a-aea1-4b5868c4d55c_EMR_1.xml doesn't exists 2016-08-13 13:48:24,821 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 61a1fcfa-ae03-450e-af12-36df6ec1e7db::EMR::1 File versions/61a1fcfa-ae03-450e-af12-36df6ec1e7db_EMR_1.xml doesn't exists 2016-08-13 13:48:24,822 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version 1c6fa300-022f-4d2b-849d-956d64cd806c::EMR::1 File versions/1c6fa300-022f-4d2b-849d-956d64cd806c_EMR_1.xml doesn't exists 2016-08-13 13:48:24,881 [quartzScheduler_Worker-3] INFO data.DataIndexerService - avoid indexing version de248ecb-94cd-43a7-b5ab-0d6a4d43bc64::EMR::1 File versions/de248ecb-94cd-43a7-b5ab-0d6a4d43bc64_EMR_1.xml doesn't exists

ppazos commented 8 years ago

the database was cleaned.