search

fli-iam / shanoir-ng

Shanoir (SHAring iN vivO Imaging Resources)
GNU General Public License v3.0
21 stars 22 forks source link

Crazy amount of logs / duplicate entry in zip #1992

Open julien-louis opened 10 months ago

julien-louis commented 10 months ago

Today during the test session the datasets logs were endless, sample : 

2023-12-14T10:23:14.205+01:00 ERROR 1 --- [nio-9904-exec-7] o.s.ng.download.WADODownloaderService    : A dicom file could not be downloaded from the pacs:
qualif-datasets-1  | 
qualif-datasets-1  | java.util.zip.ZipException: duplicate entry: actimovi_204_Exam-14387-actemovi_RS/204_20191108_RS_MB3_3x3x3_135mmFH_500dyn_1.4.9.12.34.1.8527.2020397385037063684829210808515817401781.dcm
qualif-datasets-1  |    at java.base/java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:244)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.extractDICOMZipFromMHTMLFile(WADODownloaderService.java:508)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadAndWriteFileInZip(WADODownloaderService.java:228)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadDicomFilesForURLsAsZip(WADODownloaderService.java:177)
qualif-datasets-1  |    at org.shanoir.ng.dataset.service.DatasetDownloaderServiceImpl.massiveDownload(DatasetDownloaderServiceImpl.java:313)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController.massiveDownloadByDatasetIds(DatasetApiController.java:372)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:199)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController$$SpringCGLIB$$0.massiveDownloadByDatasetIds(<generated>)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
qualif-datasets-1  |    at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
qualif-datasets-1  |    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
qualif-datasets-1  |    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
qualif-datasets-1  |    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
qualif-datasets-1  |    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
qualif-datasets-1  |    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
qualif-datasets-1  |    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
qualif-datasets-1  |    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
qualif-datasets-1  |    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
qualif-datasets-1  |    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
qualif-datasets-1  |    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
qualif-datasets-1  |    at java.base/java.lang.Thread.run(Thread.java:840)
qualif-datasets-1  | 
qualif-datasets-1  | 2023-12-14T10:23:14.205+01:00 ERROR 1 --- [nio-9904-exec-5] o.s.ng.download.WADODownloaderService    : A dicom file could not be downloaded from the pacs:
qualif-datasets-1  | 
qualif-datasets-1  | java.util.zip.ZipException: duplicate entry: actimovi_204_Exam-14387-actemovi_RS/204_20191108_RS_MB3_3x3x3_135mmFH_500dyn_1.4.9.12.34.1.8527.2020397385037063684829210808515817401781.dcm
qualif-datasets-1  |    at java.base/java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:244)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.extractDICOMZipFromMHTMLFile(WADODownloaderService.java:508)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadAndWriteFileInZip(WADODownloaderService.java:228)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadDicomFilesForURLsAsZip(WADODownloaderService.java:177)
qualif-datasets-1  |    at org.shanoir.ng.dataset.service.DatasetDownloaderServiceImpl.massiveDownload(DatasetDownloaderServiceImpl.java:313)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController.massiveDownloadByDatasetIds(DatasetApiController.java:372)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:199)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController$$SpringCGLIB$$0.massiveDownloadByDatasetIds(<generated>)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
qualif-datasets-1  |    at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
qualif-datasets-1  |    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
qualif-datasets-1  |    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
qualif-datasets-1  |    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
qualif-datasets-1  |    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
qualif-datasets-1  |    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
qualif-datasets-1  |    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
qualif-datasets-1  |    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
qualif-datasets-1  |    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
qualif-datasets-1  |    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
qualif-datasets-1  |    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
qualif-datasets-1  |    at java.base/java.lang.Thread.run(Thread.java:840)
qualif-datasets-1  | 
qualif-datasets-1  | 2023-12-14T10:23:14.209+01:00 ERROR 1 --- [nio-9904-exec-6] o.s.ng.download.WADODownloaderService    : A dicom file could not be downloaded from the pacs:
qualif-datasets-1  | 
qualif-datasets-1  | java.util.zip.ZipException: duplicate entry: actimovi_204_Exam-14387-actemovi_RS/204_20191108_RS_MB3_3x3x3_135mmFH_500dyn_1.4.9.12.34.1.8527.2020397385037063684829210808515817401781.dcm
qualif-datasets-1  |    at java.base/java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:244)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.extractDICOMZipFromMHTMLFile(WADODownloaderService.java:508)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadAndWriteFileInZip(WADODownloaderService.java:228)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadDicomFilesForURLsAsZip(WADODownloaderService.java:177)
qualif-datasets-1  |    at org.shanoir.ng.dataset.service.DatasetDownloaderServiceImpl.massiveDownload(DatasetDownloaderServiceImpl.java:313)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController.massiveDownloadByDatasetIds(DatasetApiController.java:372)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:199)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController$$SpringCGLIB$$0.massiveDownloadByDatasetIds(<generated>)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
qualif-datasets-1  |    at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
qualif-datasets-1  |    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
qualif-datasets-1  |    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
qualif-datasets-1  |    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
qualif-datasets-1  |    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
qualif-datasets-1  |    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
qualif-datasets-1  |    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
qualif-datasets-1  |    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
qualif-datasets-1  |    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
qualif-datasets-1  |    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
qualif-datasets-1  |    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
qualif-datasets-1  |    at java.base/java.lang.Thread.run(Thread.java:840)
qualif-datasets-1  | 
qualif-datasets-1  | 2023-12-14T10:23:14.213+01:00 ERROR 1 --- [nio-9904-exec-5] o.s.ng.download.WADODownloaderService    : A dicom file could not be downloaded from the pacs:
qualif-datasets-1  | 
qualif-datasets-1  | java.util.zip.ZipException: duplicate entry: actimovi_204_Exam-14387-actemovi_RS/204_20191108_RS_MB3_3x3x3_135mmFH_500dyn_1.4.9.12.34.1.8527.5048625800348892693100094687766387194414.dcm
qualif-datasets-1  |    at java.base/java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:244)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.extractDICOMZipFromMHTMLFile(WADODownloaderService.java:508)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadAndWriteFileInZip(WADODownloaderService.java:228)
qualif-datasets-1  |    at org.shanoir.ng.download.WADODownloaderService.downloadDicomFilesForURLsAsZip(WADODownloaderService.java:177)
qualif-datasets-1  |    at org.shanoir.ng.dataset.service.DatasetDownloaderServiceImpl.massiveDownload(DatasetDownloaderServiceImpl.java:313)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController.massiveDownloadByDatasetIds(DatasetApiController.java:372)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:199)
qualif-datasets-1  |    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:756)
qualif-datasets-1  |    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
qualif-datasets-1  |    at org.shanoir.ng.dataset.controler.DatasetApiController$$SpringCGLIB$$0.massiveDownloadByDatasetIds(<generated>)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
qualif-datasets-1  |    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
qualif-datasets-1  |    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
qualif-datasets-1  |    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
qualif-datasets-1  |    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
qualif-datasets-1  |    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
qualif-datasets-1  |    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
qualif-datasets-1  |    at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
qualif-datasets-1  |    at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.shanoir.ng.dicom.web.StowRSMultipartRelatedRequestFilter.doFilter(StowRSMultipartRelatedRequestFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
qualif-datasets-1  |    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
qualif-datasets-1  |    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:145)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
qualif-datasets-1  |    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
qualif-datasets-1  |    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
qualif-datasets-1  |    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
qualif-datasets-1  |    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
qualif-datasets-1  |    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
qualif-datasets-1  |    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
qualif-datasets-1  |    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
qualif-datasets-1  |    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
qualif-datasets-1  |    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
qualif-datasets-1  |    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
qualif-datasets-1  |    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
qualif-datasets-1  |    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
qualif-datasets-1  |    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
qualif-datasets-1  |    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
qualif-datasets-1  |    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
qualif-datasets-1  |    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
qualif-datasets-1  |    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
qualif-datasets-1  |    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
qualif-datasets-1  |    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
qualif-datasets-1  |    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
qualif-datasets-1  |    at java.base/java.lang.Thread.run(Thread.java:840)
qualif-datasets-1  |
julien-louis commented 10 months ago

the error message is misleading. For information ZipException extends IOException so it's catch here : 

private String downloadAndWriteFileInZip(String url, ZipOutputStream zipOutputStream, String name) throws ZipPacsFileException {
        byte[] responseBody = null;
        try {
            responseBody = downloadFileFromPACS(url);
            this.extractDICOMZipFromMHTMLFile(responseBody, extractInstanceUID(url),  name, zipOutputStream);
            return name + DCM;
        } catch (IOException | MessagingException e) {
            LOG.error("A dicom file could not be downloaded from the pacs:", e);
            throw new ZipPacsFileException(e);
        } catch (HttpClientErrorException e) {
            //LOG.error("A dicom file could not be downloaded from the pacs:", e);
            throw new ZipPacsFileException("received " + e.getStatusCode() + " from PACS", e);
        }
    }
julien-louis commented 10 months ago

is it normal that several entries in dataset_file have the exact same path ? 

mysql@shanoir-2016 qualif-datasets> select * from dataset_file where path like "%1.4.9.12.34.1.8527.5048625800348892693100094687766387194414%";
+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| id       | pacs | path                                                                                                                                                                                                                                                                                                                | dataset_expression_id |
+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
| 21997930 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 21999978 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22002026 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22004074 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22006122 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22008170 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22010218 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22012266 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22014314 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22016362 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
| 22018410 |     | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4336744224378007316674743989742922429095&seriesUID=1.4.9.12.34.1.8527.7030926963636721255171582706588720155772&objectUID=1.4.9.12.34.1.8527.5048625800348892693100094687766387194414&contentType=application/dicom |                764157 |
+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+
11 rows in set (1 min 40,527 sec)
julien-louis commented 10 months ago

They all come from dataset 385105 : 

mysql@shanoir-2016 qualif-datasets> select ds.id from dataset ds, dataset_expression dse, dataset_file dsf where dsf.id in (21997930,21999978,22002026,22004074,22006122,22008170,22010218,22012266,22014314,22016362,22018410) and dsf.dataset_expression_id = dse.id and dse.dataset_id = ds.id;
+--------+
| id     |
+--------+
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
| 385105 |
+--------+
11 rows in set (0,048 sec)
julien-louis commented 10 months ago

The real problem ? : 

select count(dse.id) from dataset_expression dse, dataset ds, dataset_acquisition dsa, examination e where e.study_id = 212 and e.id = dsa.examination_id and dsa.id = ds.dataset_acquisition_id and ds.id = dse.dataset_id;
+---------------+
| count(dse.id) |
+---------------+
|            32 |
+---------------+
1 row in set (0,011 sec)

mysql@shanoir-2016 qualif-datasets> select count(dsf.id) from dataset_file dsf, dataset_expression dse, dataset ds, dataset_acquisition dsa, examination e where e.study_id = 212 and e.id = dsa.examination_id and dsa.id = ds.dataset_acquisition_id and ds.id = dse.dataset_id and dse.id = dsf.dataset_expression_id;
+---------------+
| count(dsf.id) |
+---------------+
|        158576 |
+---------------+
1 row in set (0,103 sec)

=> there are 158576 dataset_file in this 16 datasets study, and I believe that most are duplicates

julien-louis commented 10 months ago

grouping the duplicates : select count(dsf.id) as nb_duplicates, path from dataset_file dsf, dataset_expression dse, dataset ds, dataset_acquisition dsa, examination e where e.study_id = 212 and e.id = dsa.examination_id and dsa.id = ds.dataset_acquisition_id and ds.id = dse.dataset_id and dse.id = dsf.dataset_expression_id group by dsf.path order by nb_duplicates asc;

result in 15412 rows (that's the real number of dataset_files in this study), the trail looking like that : 

|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.4096556163597913295929812890132171953235&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.7164425706276273673107239730990312406322&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.4008884918348032639321442073516111688046&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.4006537130127081708919460127079985344758&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.7121370909008417111683432736770527809779&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.6323194129835014793803230299135453218562&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.1023689849521046522610737473131764845055&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.6089091733105030839652610567622075450692&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.6068484676394264123703158497835675990921&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.4165189904090647034808299647492942050548&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.3275082130928008578323661818583501050965&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.9195507601804765448188476350367750184684&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.1339449194444259404184301304274413632813&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.6327631773525496804100342789428161187060&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.7165945934742299770385666078889615303275&contentType=application/dicom |
|            11 | http://pacs:8080/dcm4chee-arc/aets/AS_RECEIVED/wado?requestType=WADO&studyUID=1.4.9.12.34.1.8527.4246182187721076211291425927864605197657&seriesUID=1.4.9.12.34.1.8527.4295261971745855258436035383143019615651&objectUID=1.4.9.12.34.1.8527.3051577642581144571885550635436081904709&contentType=application/dicom |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

So most of the dataset_files are duplicate 11 times.

I will make a script that cleans it up but first I will try to execute a script on the whole table to detect other duplicates, if it's not too long to execute !

julien-louis commented 10 months ago 
mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 0 and dsf.dataset_expression_id <= 100000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                 73863 |
|                 72353 |
|                 72369 |
|                 72379 |
|                 72381 |
|                 72383 |
|                 72385 |
|                 72413 |
|                 72415 |
|                 72375 |
|                 72377 |
|                 72389 |
|                 72391 |
|                 72393 |
|                 72457 |
|                 72459 |
|                 72481 |
|                 72469 |
|                 72491 |
|                 72511 |
|                 72461 |
|                 72483 |
|                 72503 |
|                 72465 |
|                 72487 |
|                 72507 |
|                 72467 |
|                 72489 |
|                 72509 |
|                 72463 |
|                 72485 |
|                 72505 |
|                 72501 |
|                 72521 |
|                 72493 |
|                 72513 |
|                 72497 |
|                 72517 |
|                 72499 |
|                 72519 |
|                 72495 |
|                 72515 |
|                 73205 |
|                 73203 |
|                 73343 |
|                 73295 |
|                 73301 |
|                 73273 |
|                 73275 |
|                 73267 |
|                 73269 |
|                 73271 |
+-----------------------+
52 rows in set (4 min 11,488 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 100000 and dsf.dataset_expression_id <= 150000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                119312 |
|                140850 |
|                140852 |
|                140858 |
|                144998 |
|                145002 |
|                104923 |
|                110546 |
|                123990 |
|                123988 |
|                125641 |
|                125640 |
|                119885 |
+-----------------------+
13 rows in set (40,623 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 150000 and dsf.dataset_expression_id <= 200000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                180795 |
|                184488 |
|                184494 |
+-----------------------+
3 rows in set (41,870 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 200000 and dsf.dataset_expression_id <= 250000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                202877 |
|                207934 |
|                216925 |
+-----------------------+
3 rows in set (48,402 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 250000 and dsf.dataset_expression_id <= 300000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
Empty set (30,971 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 350000 and dsf.dataset_expression_id <= 400000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                395965 |
|                395969 |
|                395977 |
|                395967 |
|                395971 |
|                395979 |
|                395963 |
|                395975 |
|                396883 |
|                397455 |
|                397453 |
|                354174 |
|                364681 |
|                364709 |
+-----------------------+
14 rows in set (37,450 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 400000 and dsf.dataset_expression_id <= 450000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                437128 |
|                422979 |
|                422981 |
|                422983 |
|                422985 |
|                422987 |
|                423039 |
|                423015 |
|                422991 |
|                423023 |
|                423045 |
|                423027 |
|                423003 |
|                422999 |
|                423021 |
|                423037 |
|                422989 |
|                423041 |
|                423019 |
|                423011 |
|                422993 |
|                423017 |
|                423025 |
|                423005 |
|                422997 |
|                423007 |
|                423029 |
|                423009 |
|                423001 |
|                423035 |
|                422995 |
|                423013 |
|                423033 |
|                423031 |
|                423043 |
|                423047 |
|                423049 |
|                423051 |
|                423055 |
|                423053 |
|                423057 |
|                423059 |
|                423111 |
|                423087 |
|                423063 |
|                423095 |
|                423117 |
|                423099 |
|                423075 |
|                423071 |
|                423065 |
|                423109 |
|                423061 |
|                423113 |
|                423091 |
|                423083 |
|                423093 |
|                423089 |
|                423097 |
|                423077 |
|                423069 |
|                423079 |
|                423103 |
|                423081 |
|                423073 |
|                423107 |
|                423067 |
|                423085 |
|                423105 |
|                423101 |
|                423115 |
|                423119 |
|                423121 |
|                423123 |
|                423125 |
|                423129 |
|                423127 |
|                423133 |
|                423131 |
|                423135 |
+-----------------------+
80 rows in set (1 min 17,684 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 450000 and dsf.dataset_expression_id <= 500000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                478168 |
|                454518 |
+-----------------------+
2 rows in set (43,792 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 500000 and dsf.dataset_expression_id <= 550000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
Empty set (1 min 22,462 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 550000 and dsf.dataset_expression_id <= 600000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                574106 |
|                561601 |
|                596731 |
+-----------------------+
3 rows in set (1 min 22,686 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 600000 and dsf.dataset_expression_id <= 650000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                628024 |
|                628026 |
|                628028 |
+-----------------------+
3 rows in set (31,704 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 650000 and dsf.dataset_expression_id <= 700000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                672441 |
|                660906 |
+-----------------------+
2 rows in set (32,205 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 700000 and dsf.dataset_expression_id <= 750000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
Empty set (9,757 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 750000 and dsf.dataset_expression_id <= 800000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                759065 |
|                759073 |
|                759071 |
|                759063 |
|                759069 |
|                758053 |
|                758079 |
|                759011 |
|                758997 |
|                759005 |
|                759003 |
|                758999 |
|                758055 |
|                764189 |
|                782579 |
|                759017 |
|                759013 |
|                759019 |
|                759021 |
|                759029 |
|                758127 |
|                764197 |
|                759047 |
|                759049 |
|                759055 |
|                759061 |
|                759053 |
|                767115 |
|                764157 |
|                758023 |
|                767155 |
|                767163 |
|                767157 |
|                767151 |
|                767149 |
|                764143 |
|                759045 |
|                767131 |
|                767125 |
|                767117 |
|                767119 |
|                767123 |
|                759121 |
|                759115 |
|                759123 |
|                759113 |
|                759119 |
|                759109 |
|                759111 |
|                759105 |
|                759107 |
|                759101 |
|                758979 |
|                764365 |
|                758985 |
|                758991 |
|                758995 |
|                758993 |
|                758989 |
|                767141 |
|                767135 |
|                767143 |
|                767133 |
|                767139 |
|                759081 |
|                759091 |
|                759089 |
|                759085 |
|                759087 |
|                759079 |
|                758133 |
|                764207 |
|                758091 |
+-----------------------+
73 rows in set (3 min 58,976 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 800000 and dsf.dataset_expression_id <= 850000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
Empty set (4 min 26,038 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 850000 and dsf.dataset_expression_id <= 900000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
Empty set (2 min 11,734 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 900000 and dsf.dataset_expression_id <= 950000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
Empty set (30,930 sec)

mysql@shanoir-2016 qualif-datasets> select distinct dsf.dataset_expression_id from dataset_file dsf where dsf.dataset_expression_id > 950000 group by dsf.path, dsf.dataset_expression_id having count(dsf.id) > 1;
+-----------------------+
| dataset_expression_id |
+-----------------------+
|                956602 |
|                956604 |
|                956606 |
+-----------------------+
julien-louis commented 10 months ago

There are 928964 dataset files in dataset_expressions that have multiple dataset_files. (the other dataset_expressions have only one file)

mysql@shanoir-2016 qualif-datasets> create view dataset_file_tmp as select * from dataset_file where dataset_expression_id in (73863, 72353, 72369, 72379, 72381, 72383, 72385, 72413, 72415, 72375, 72377, 72389,
72391, 72393, 72457, 72459, 72481, 72469, 72491, 72511, 72461, 72483, 72503, 72465, 72487, 72507, 72467, 72489, 72509, 72463, 72485, 72505, 72501, 72521, 72493, 72513, 72497, 72517, 72499, 72519, 72495, 72515, 73205, 73203, 73343, 73295, 73301, 73273, 73275, 73267, 73269, 73271, 119312, 140850, 140852, 140858, 144998, 145002, 104923, 110546, 123990, 123988, 125641, 125640, 119885, 180795, 184488, 184494, 202877, 207934, 216925, 395965, 395969, 395977, 395967, 395971, 395979, 395963, 395975, 396883, 397455, 397453, 354174, 364681, 364709, 437128, 422979, 422981, 422983, 422985, 422987, 423039, 423015, 422991, 423023, 423045, 423027, 423003, 422999, 423021, 423037, 422989, 423041, 423019, 423011, 422993, 423017, 423025, 423005, 422997, 423007, 423029, 423009, 423001, 423035, 422995, 423013, 423033, 423031, 423043, 423047, 423049, 423051, 423055, 423053, 423057, 423059, 423111, 423087, 423063, 423095, 423117, 423099, 423075, 423071, 423065, 423109, 423061, 423113, 423091, 423083, 423093, 423089, 423097, 423077, 423069, 423079, 423103, 423081, 423073, 423107, 423067, 423085, 423105, 423101, 423115, 423119, 423121, 423123, 423125, 423129, 423127, 423133, 423131, 423135, 478168, 454518, 574106, 561601, 596731, 628024, 628026, 628028, 672441, 660906, 759065, 759073, 759071, 759063, 759069, 758053, 758079, 759011, 758997, 759005, 759003, 758999, 758055, 764189, 782579, 759017, 759013, 759019, 759021, 759029, 758127, 764197, 759047, 759049, 759055, 759061, 759053, 767115, 764157, 758023, 767155, 767163, 767157, 767151, 767149, 764143, 759045, 767131, 767125, 767117, 767119, 767123, 759121, 759115, 759123, 759113, 759119, 759109, 759111, 759105, 759107, 759101, 758979,
764365, 758985, 758991, 758995, 758993, 758989, 767141, 767135, 767143, 767133, 767139, 759081, 759091, 759089, 759085, 759087, 759079, 758133, 764207, 758091, 956602, 956604, 956606);
Query OK, 0 rows affected (0,003 sec)

mysql@shanoir-2016 qualif-datasets> select count(*) from dataset_file_tmp;
+----------+
| count(*) |
+----------+
|   928694 |
+----------+
1 row in set (1,026 sec)
julien-louis commented 10 months ago

(251 dataset_expressions have several files, 921826 have just one) : 

mysql@shanoir-2016 qualif-datasets> select count(*) from dataset_expression where id in (73863, 72353, 72369, 72379, 72381, 72383, 72385, 72413, 72415, 72375, 72377, 72389, 72391, 72393, 72457, 72459, 72481, 72469, 72491, 72511, 72461, 72483, 72503, 72465, 72487, 72507, 72467, 72489, 72509, 72463, 72485, 72505, 72501, 72521, 72493, 72513, 72497, 72517, 72499, 72519, 72495, 72515, 73205, 73203, 73343, 73295, 73301, 7327                                                                                                                                                                                               395969, 395977, 3959                                                                                                                                                                                               999, 423021, 423037,                                                                                                                                                                                               , 423057, 423059, 423111, 423087, 423063, 423095, 423117, 423099, 423075, 423071, 423065, 423109, 423061, 423113, 423091, 423083, 423093, 423089, 423097, 423077, 423069, 423079, 423103, 423081, 423073, 423107, 423067, 423085, 423105, 423101, 423115, 423119, 423121, 423123, 423125, 423129, 423127, 423133, 423131, 423135, 478168, 454518, 574106, 561601, 596731, 628024, 628026, 628028, 672441, 660906, 759065, 759073, 759071, 759063, 759069,
758053, 758079, 759011, 758997, 759005, 759003, 758999, 758055, 764189, 782579, 759017, 759013, 759019, 759021, 759029, 758127, 764197, 759047, 759049, 759055, 759061, 759053, 767115, 764157, 758023, 767155, 767                                                                                                                                                                                               8991, 758995, 758993,
+----------+
| count(*) |
+----------+
|      251 |
+----------+
1 row in set (0,154 sec)

mysql@shanoir-2016 qualif-datasets> select count(*) from dataset_expression where id not in (73863, 72353, 72369, 72379, 72381, 72383, 72385, 72413, 72415, 72375, 72377, 72389, 72391, 72393, 72457, 72459, 72481,
 72469, 72491, 72511, 72461, 72483, 72503, 72465, 72487, 72507, 72467, 72489, 72509, 72463, 72485, 72505, 72501, 72521, 72493, 72513, 72497, 72517, 72499, 72519, 72495, 72515, 73205, 73203, 73343, 73295, 73301,
73273, 73275, 73267, 73269, 73271, 119312, 140850, 140852, 140858, 144998, 145002, 104923, 110546, 123990, 123988, 125641, 125640, 119885, 180795, 184488, 184494, 202877, 207934, 216925, 395965, 395969, 395977,
395967, 395971, 395979, 395963, 395975, 396883, 397455, 397453, 354174, 364681, 364709, 437128, 422979, 422981, 422983, 422985, 422987, 423039, 423015, 422991, 423023, 423045, 423027, 423003, 422999, 423021, 423037, 422989, 423041, 423019, 423011, 422993, 423017, 423025, 423005, 422997, 423007, 423029, 423009, 423001, 423035, 422995, 423013, 423033, 423031, 423043, 423047, 423049, 423051, 423055, 423053, 423057, 423059
, 423111, 423087, 423063, 423095, 423117, 423099, 423075, 423071, 423065, 423109, 423061, 423113, 423091, 423083, 423093, 423089, 423097, 423077, 423069, 423079, 423103, 423081, 423073, 423107, 423067, 423085, 423105, 423101, 423115, 423119, 423121, 423123, 423125, 423129, 423127, 423133, 423131, 423135, 478168, 454518, 574106, 561601, 596731, 628024, 628026, 628028, 672441, 660906, 759065, 759073, 759071, 759063, 7590
69, 758053, 758079, 759011, 758997, 759005, 759003, 758999, 758055, 764189, 782579, 759017, 759013, 759019, 759021, 759029, 758127, 764197, 759047, 759049, 759055, 759061, 759053, 767115, 764157, 758023, 767155,
 767163, 767157, 767151, 767149, 764143, 759045, 767131, 767125, 767117, 767119, 767123, 759121, 759115, 759123, 759113, 759119, 759109, 759111, 759105, 759107, 759101, 758979, 764365, 758985, 758991, 758995, 758993
+----------+
| count(*) |
+----------+
|   921826 |
+----------+
1 row in set (0,892 sec)
julien-louis commented 10 months ago

Among those 251, most have 2 to 4 files, but some have thousands ! : 

mysql@shanoir-2016 qualif-datasets> select dataset_expression_id, count(*) from dataset_file_tmp group by dataset_expression_id;
+-----------------------+----------+
| dataset_expression_id | count(*) |
+-----------------------+----------+
|                 72353 |        2 |
|                 72369 |        2 |
|                 72375 |        2 |
|                 72377 |        2 |
|                 72379 |        2 |
|                 72381 |        2 |
|                 72383 |        2 |
|                 72385 |        2 |
|                 72389 |        2 |
|                 72391 |        3 |
|                 72393 |        3 |
|                 72413 |        2 |
|                 72415 |        2 |
|                 72457 |        2 |
|                 72459 |        3 |
|                 72461 |        4 |
|                 72463 |        4 |
|                 72465 |        4 |
|                 72467 |        4 |
|                 72469 |        4 |
|                 72481 |        3 |
|                 72483 |        4 |
|                 72485 |        4 |
|                 72487 |        4 |
|                 72489 |        4 |
|                 72491 |        4 |
|                 72493 |        3 |
|                 72495 |        3 |
|                 72497 |        3 |
|                 72499 |        3 |
|                 72501 |        3 |
|                 72503 |        4 |
|                 72505 |        4 |
|                 72507 |        4 |
|                 72509 |        4 |
|                 72511 |        4 |
|                 72513 |        3 |
|                 72515 |        3 |
|                 72517 |        3 |
|                 72519 |        3 |
|                 72521 |        3 |
|                 73203 |        2 |
|                 73205 |        2 |
|                 73267 |        2 |
|                 73269 |        2 |
|                 73271 |        2 |
|                 73273 |        2 |
|                 73275 |        2 |
|                 73295 |        2 |
|                 73301 |        2 |
|                 73343 |      200 |
|                 73863 |        2 |
|                104923 |        2 |
|                110546 |        2 |
|                119312 |        2 |
|                119885 |    14400 |
|                123988 |    14393 |
|                123990 |     4271 |
|                125640 |     4271 |
|                125641 |    14400 |
|                140850 |        2 |
|                140852 |        2 |
|                140858 |        2 |
|                144998 |        2 |
|                145002 |        2 |
|                180795 |        2 |
|                184488 |        2 |
|                184494 |        2 |
|                202877 |        2 |
|                207934 |        2 |
|                216925 |        2 |
|                354174 |     2311 |
|                364681 |     2311 |
|                364709 |     2311 |
|                395963 |        2 |
|                395965 |        4 |
|                395967 |        4 |
|                395969 |        4 |
|                395971 |        4 |
|                395975 |        2 |
|                395977 |        4 |
|                395979 |        4 |
|                396883 |        2 |
|                397453 |        2 |
|                397455 |        2 |
|                422979 |      190 |
|                422981 |      352 |
|                422983 |      160 |
|                422985 |      352 |
|                422987 |        2 |
|                422989 |        2 |
|                422991 |        2 |
|                422993 |        2 |
|                422995 |        2 |
|                422997 |        2 |
|                422999 |        2 |
|                423001 |        2 |
|                423003 |        2 |
|                423005 |        2 |
|                423007 |        2 |
|                423009 |        2 |
|                423011 |        2 |
|                423013 |        2 |
|                423015 |        2 |
|                423017 |        2 |
|                423019 |        2 |
|                423021 |        2 |
|                423023 |        2 |
|                423025 |        2 |
|                423027 |        2 |
|                423029 |        2 |
|                423031 |        2 |
|                423033 |        2 |
|                423035 |        2 |
|                423037 |        2 |
|                423039 |        2 |
|                423041 |        2 |
|                423043 |        2 |
|                423045 |        2 |
|                423047 |        2 |
|                423049 |      140 |
|                423051 |      140 |
|                423053 |      140 |
|                423055 |      140 |
|                423057 |      140 |
|                423059 |        2 |
|                423061 |        2 |
|                423063 |        2 |
|                423065 |        2 |
|                423067 |        2 |
|                423069 |        2 |
|                423071 |        2 |
|                423073 |        2 |
|                423075 |        2 |
|                423077 |        2 |
|                423079 |        2 |
|                423081 |        2 |
|                423083 |        2 |
|                423085 |        2 |
|                423087 |        2 |
|                423089 |        2 |
|                423091 |        2 |
|                423093 |        2 |
|                423095 |        2 |
|                423097 |        2 |
|                423099 |        2 |
|                423101 |        2 |
|                423103 |        2 |
|                423105 |        2 |
|                423107 |        2 |
|                423109 |        2 |
|                423111 |        2 |
|                423113 |        2 |
|                423115 |        2 |
|                423117 |        2 |
|                423119 |        2 |
|                423121 |      140 |
|                423123 |      140 |
|                423125 |      140 |
|                423127 |      140 |
|                423129 |      140 |
|                423131 |      100 |
|                423133 |      100 |
|                423135 |      100 |
|                437128 |        2 |
|                454518 |        2 |
|                478168 |        2 |
|                561601 |       97 |
|                574106 |        2 |
|                596731 |      344 |
|                628024 |        2 |
|                628026 |        2 |
|                628028 |        2 |
|                660906 |        4 |
|                672441 |        2 |
|                758023 |    22500 |
|                758053 |    22500 |
|                758055 |    22500 |
|                758079 |    22500 |
|                758091 |    22500 |
|                758127 |    22500 |
|                758133 |    22500 |
|                758979 |    16020 |
|                758985 |     5940 |
|                758989 |     5940 |
|                758991 |     5940 |
|                758993 |     5940 |
|                758995 |    22500 |
|                758997 |     5940 |
|                758999 |     5940 |
|                759003 |     5940 |
|                759005 |     5940 |
|                759011 |    22500 |
|                759013 |     5940 |
|                759017 |     5940 |
|                759019 |     5940 |
|                759021 |     5940 |
|                759029 |    22500 |
|                759045 |    22500 |
|                759047 |     5940 |
|                759049 |     5940 |
|                759053 |     5940 |
|                759055 |     5940 |
|                759061 |    22500 |
|                759063 |     5940 |
|                759065 |     5940 |
|                759069 |     5940 |
|                759071 |     5940 |
|                759073 |    22500 |
|                759079 |     5940 |
|                759081 |     5940 |
|                759085 |     5940 |
|                759087 |     5940 |
|                759089 |     2949 |
|                759091 |    22500 |
|                759101 |     5940 |
|                759105 |     5940 |
|                759107 |     5940 |
|                759109 |     5940 |
|                759111 |    22500 |
|                759113 |     5940 |
|                759115 |     5940 |
|                759119 |     5940 |
|                759121 |     5940 |
|                759123 |    22500 |
|                764143 |    22500 |
|                764157 |    22500 |
|                764189 |    22500 |
|                764197 |    22500 |
|                764207 |    22500 |
|                764365 |    22500 |
|                767115 |    22500 |
|                767117 |     5940 |
|                767119 |     5940 |
|                767123 |     5940 |
|                767125 |     5940 |
|                767131 |    22500 |
|                767133 |     5940 |
|                767135 |     5940 |
|                767139 |     5940 |
|                767141 |     5940 |
|                767143 |    22500 |
|                767149 |     5940 |
|                767151 |     5940 |
|                767155 |     5940 |
|                767157 |     5940 |
|                767163 |    22500 |
|                782579 |      166 |
|                956602 |      320 |
|                956604 |      144 |
|                956606 |      320 |
+-----------------------+----------+
251 rows in set (0,496 sec)
julien-louis commented 10 months ago

what dataset expression format ?

mysql@shanoir-2016 qualif-datasets> select count(id), dataset_expression_format from dataset_expression where id in (73863, 72353, 72369, 72379, 72381, 72383, 72385, 72413, 72415, 72375, 72377, 72389, 72391, 72393, 72457, 72459, 72481, 72469, 72491, 72511, 72461, 72483, 72503, 72465, 72487, 72507, 72467, 72489, 72509, 72463, 72485, 72505, 72501, 72521, 72493, 72513, 72497, 72517, 72499, 72519, 72495, 72515, 73205, 73203, 73343, 73295, 73301, 73273, 73275, 73267, 73269, 73271, 119312, 140850, 140852, 140858, 144998, 145002, 104923, 110546, 123990, 123988, 125641, 125640, 119885, 180795, 184488, 184494, 202877, 207934, 216925, 395965, 395969, 395977, 395967, 395971, 395979, 395963, 395975, 396883, 397455, 397453, 354174, 364681, 364709, 437128, 422979, 422981, 422983, 422985, 422987, 423039, 423015, 422991, 423023, 423045, 423027, 423003, 422999, 423021, 423037, 422989, 423041, 423019, 423011, 422993, 423017, 423025, 423005, 422997, 423007, 423029, 423009, 423001, 423035, 422995, 423013, 423033, 423031, 423043, 423047, 423049, 423051, 423055, 423053, 423057, 423059, 423111, 423087, 423063, 423095, 423117, 423099, 423075, 423071, 423065, 423109, 423061, 423113, 423091, 423083, 423093, 423089, 423097, 423077, 423069, 423079, 423103, 423081, 423073, 423107, 423067, 423085, 423105, 423101, 423115, 423119, 423121, 423123, 423125, 423129, 423127, 423133, 423131, 423135, 478168, 454518, 574106, 561601, 596731, 628024, 628026, 628028, 672441, 660906, 759065, 759073, 759071, 759063, 759069, 758053, 758079, 759011, 758997, 759005, 759003, 758999, 758055, 764189, 782579, 759017, 759013, 759019, 759021, 759029, 758127, 764197, 759047, 759049, 759055, 759061, 759053, 767115, 764157, 758023, 767155, 767163, 767157, 767151, 767149, 764143, 759045, 767131, 767125, 767117, 767119, 767123, 759121, 759115, 759123, 759113, 759119, 759109, 759111, 759105, 759107, 759101, 758979, 764365, 758985, 758991, 758995, 758993, 758989, 767141, 767135, 767143, 767133, 767139, 759081, 759091, 759089, 759085, 759087, 759079, 758133, 764207, 758091, 956602, 956604, 956606) group by dataset_expression_format;
+-----------+---------------------------+
| count(id) | dataset_expression_format |
+-----------+---------------------------+
|        85 |                         2 | (NIFTI_SINGLE_FILE)
|       166 |                         6 | (DICOM)
+-----------+---------------------------+
2 rows in set (0,003 sec)

mysql@shanoir-2016 qualif-datasets> select count(id), dataset_expression_format from dataset_expression group by dataset_expression_format;
+-----------+---------------------------+
| count(id) | dataset_expression_format |
+-----------+---------------------------+
|    451435 |                         2 | (NIFTI_SINGLE_FILE)
|    470094 |                         6 | (DICOM)
|         9 |                        17 | (SIEMENS_SPECTROSCOPY)
|        11 |                        18 | (PHILIPS_SPECTROSCOPY)
|       246 |                        19 | (EEG)
|       282 |                        20 | (BIDS)
+-----------+---------------------------+
6 rows in set (1,023 sec)
julien-louis commented 10 months ago

studies that have duplicates and their nb of dataset_files : 

mysql@shanoir-2016 qualif-datasets> select e.study_id, count(dsf.id) as nb_files from dataset_file_tmp dsf, dataset_expression dse, dataset ds, dataset_acquisition dsa, examination e, study s where e.id = dsa.examination_id and dsa.id = ds.dataset_acquisition_id and ds.id = dse.dataset_id and dse.id = dsf.dataset_expression_id and e.study_id = s.id group by e.study_id;
+----------+----------+
| study_id | nb_files |
+----------+----------+
|       11 |      346 |
|       34 |        2 |
|       39 |        4 |
|       46 |    51737 |
|       61 |       22 |
|       80 |        6 |
|      100 |     9811 |
|      121 |        2 |
|      124 |      479 |
|      170 |   707829 |
|      175 |      166 |
|      188 |        2 |
|      190 |        4 |
|      212 |   157500 |
|      241 |      784 |
julien-louis commented 10 months ago

I've added logs to my pr : 

qualif-datasets-1  | 2024-01-08T11:29:25.927+01:00  WARN 1 --- [           main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
qualif-datasets-1  | 2024-01-08T11:32:00.058+01:00 ERROR 1 --- [io-9904-exec-10] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°385105, they were ignored.
qualif-datasets-1  | 2024-01-08T11:32:00.901+01:00 ERROR 1 --- [nio-9904-exec-5] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°385098, they were ignored.
qualif-datasets-1  | 2024-01-08T11:32:06.304+01:00 ERROR 1 --- [nio-9904-exec-9] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°385121, they were ignored.
qualif-datasets-1  | 2024-01-08T11:32:08.017+01:00 ERROR 1 --- [nio-9904-exec-1] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°385125, they were ignored.
qualif-datasets-1  | 2024-01-08T11:32:36.301+01:00 ERROR 1 --- [nio-9904-exec-8] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°385130, they were ignored.
qualif-datasets-1  | 2024-01-08T11:32:40.602+01:00 ERROR 1 --- [nio-9904-exec-5] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°385209, they were ignored.
qualif-datasets-1  | 2024-01-08T11:32:46.028+01:00 ERROR 1 --- [nio-9904-exec-4] o.s.ng.download.WADODownloaderService    : There were 20452 duplicate dataset_files when zipping dataset n°386584, they were ignored.
julien-louis commented 10 months ago

These datasets (from study a**i, id : 212) are all named like "RS_MB3_3x3x3_135mmFH_500dyn"

julien-louis commented 10 months ago

equipments invloved : 

mysql@shanoir-2016 qualif-datasets> select s.id as study, ae.name, dsa.acquisition_equipment_id, count(dsf.id) as nb_files from dataset_file_tmp dsf, dataset_expression dse, dataset ds, dataset_acquisition dsa,
examination e, study s, acquisition_equipment ae where e.id = dsa.examination_id and dsa.id = ds.dataset_acquisition_id and ds.id = dse.dataset_id and dse.id = dsf.dataset_expression_id and e.study_id = s.id and ae.id = dsa.acquisition_equipment_id group by dsa.acquisition_equipment_id, s.id;
+-------+--------------------------------------------------------------------------+--------------------------+----------+
| study | name                                                                     | acquisition_equipment_id | nb_files |
+-------+--------------------------------------------------------------------------+--------------------------+----------+
|    11 | SIEMENS - Verio (3T) 40296 CHU Rennes                                    |                        1 |      346 |
|    39 | SIEMENS - Verio (3T) 40296 CHU Rennes                                    |                        1 |        4 |
|    61 | SIEMENS - Verio (3T) 40296 CHU Rennes                                    |                        1 |       22 |
|   100 | Philips Medical Systems - Achieva (3T) 17569 HCL - NeuroCardio           |                        6 |     4622 |
|    46 | Philips Medical Systems - Achieva-dStream (3T) 38235 CHU Grenoble        |                        8 |        2 |
|   170 | Philips Medical Systems - Achieva-dStream (3T) 38235 CHU Grenoble        |                        8 |   707829 |
|   190 | SIEMENS - Sonata (1.5T) 21221 CHU Nantes                                 |                        9 |        4 |
|   212 | Philips Medical Systems - Achieva (3T) 38058 GIN                         |                       13 |   157500 |
|    46 | Philips Medical Systems - Achieva (3T) 731041 CHU Toulouse               |                       24 |    51735 |
|   100 | Philips Medical Systems - Achieva (3T) 60173981 HCL - NeuroCardio        |                       46 |     2311 |
|   121 | SIEMENS - Aera (1.5T) 41575 CHU Rennes                                   |                       64 |        2 |
|   124 | SIEMENS - Aera (1.5T) 41987 CHU Nantes                                   |                       66 |       38 |
|   124 | SIEMENS - Prisma (3T) 66062 CHU St-Etienne                               |                      107 |       97 |
|   100 | SIEMENS - Skyra (3T) 46219 Hopital d'Instruction des Armées Sainte Anne  |                      128 |     2878 |
|   241 | SIEMENS - Skyra (3T) M037353 Hopital Pitié Salpétrière                   |                      144 |      784 |
|   124 | SIEMENS - Aera (1.5T) 141228 CHD de Vendée                               |                      169 |      344 |
|    34 | SIEMENS - Prisma (3T) 166066 CHU Rennes                                  |                      183 |        2 |
|   188 | SIEMENS - Prisma (3T) 166066 CHU Rennes                                  |                      183 |        2 |
|    80 | SIEMENS - Avanto (1.5T) 26932 GHBS Lorient                               |                      191 |        6 |
|   175 | Philips - IQon Spectral CT60019 HCL - CHLS                               |                      252 |      166 |
+-------+--------------------------------------------------------------------------+--------------------------+----------+
20 rows in set (1,166 sec)
michaelkain commented 10 months ago

Hi @julien-louis, thank you for your findings above. I proposed a meeting slot to discuss about it.

michaelkain commented 10 months ago

251 from 923.947 dataset_expressions concerned: 0,2% 921.826 from 29.457.913 dataset_files concerned: 3,1%