search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
204 stars 80 forks source link

Creating a version of an imported extension fails #37

Closed dkincaid closed 5 years ago

dkincaid commented 5 years ago

I am trying to import the SNOMED Veterinary Extensions into snowstorm. I followed the instructions in the "updating-snomed-and-externsions.md" document to load the extension and that went smoothly except that it didn't create a version. So I tried to create a version using the /codesystems/{shortName}/versions endpoint and that throws an exception using the input:

{
  "description": "SNOMED Veterinary extension April 2019 release",
  "effectiveDate": 20190401
}

Here is the exception:

Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/es-member/_bulk?timeout=1m], status line [HTTP/1.1 413 Request Entity Too Large]

    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:355) ~[elasticsearch-rest-client-6.0.1.jar!/:6.0.1]
    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:344) ~[elasticsearch-rest-client-6.0.1.jar!/:6.0.1]
    at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) ~[httpcore-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:439) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:329) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    ... 1 common frames omitted
kaicode commented 5 years ago

Hi @dkincaid, thanks for getting in touch. It looks like Snowstorm made a request to Elasticsearch which was too big to process. Unfortunately the part of Snowstorm code which made this request has not shown up in the stack trace there.

Can I ask, did the RF2 content you imported have the effectiveTime set in all rows or was some of the content unversioned? I'm just trying to build a picture of your scenario. If you are using a published version of the Veterinary extension where might I find a copy?

Kind regards, Kai

dkincaid commented 5 years ago

Thanks for the quick response. The whole stack trace is extremely long, but I'll paste it in at the bottom of this comment along with the log messages from the start of the manual version creation. As far as I can tell the effective time is populated in the extension files. The extension can be found here - https://vtsl.vetmed.vt.edu/extension/

2019-04-08 12:37:58.501  INFO 31343 --- [nio-8080-exec-9] o.s.s.c.data.services.CodeSystemService  : Creating Code System version - Code System: SNOMEDCT-VET, Version: 2019-04-01, Release Branch: MAIN/SNOMEDCT-VET/2019-04-01
2019-04-08 12:37:58.501  INFO 31343 --- [nio-8080-exec-9] o.s.s.c.data.services.CodeSystemService  : Versioning content...
2019-04-08 12:37:58.624  INFO 31343 --- [nio-8080-exec-9] io.kaicode.elasticvc.api.BranchService   : Open commit on MAIN/SNOMEDCT-VET at 1554745078624
2019-04-08 12:37:59.191  INFO 31343 --- [nio-8080-exec-9] i.k.elasticvc.api.ComponentService       : Saving batch of 35072 Concepts
2019-04-08 12:38:04.131  INFO 31343 --- [nio-8080-exec-9] i.k.elasticvc.api.ComponentService       : Saving batch of 91846 Descriptions
2019-04-08 12:38:17.666  INFO 31343 --- [nio-8080-exec-9] i.k.elasticvc.api.ComponentService       : Saving batch of 57208 Relationships
2019-04-08 12:38:31.187  INFO 31343 --- [nio-8080-exec-9] i.k.elasticvc.api.ComponentService       : Saving batch of 237384 ReferenceSetMembers
2019-04-08 12:39:19.923  INFO 31343 --- [nio-8080-exec-9] io.kaicode.elasticvc.api.BranchService   : Rolling back commit on MAIN/SNOMEDCT-VET started at 1554745078624
2019-04-08 12:39:25.283 ERROR 31343 --- [nio-8080-exec-9] o.s.s.rest.config.RestControllerAdvice   : Unhandled exception.

org.elasticsearch.ElasticsearchStatusException: Unable to parse response body
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:538) ~[elasticsearch-rest-high-level-client-6.0.1.jar!/:6.0.1]
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:441) ~[elasticsearch-rest-high-level-client-6.0.1.jar!/:6.0.1]
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:414) ~[elasticsearch-rest-high-level-client-6.0.1.jar!/:6.0.1]
    at org.springframework.data.elasticsearch.rest.ESRestHighLevelClient.performRequestAndParseEntity(ESRestHighLevelClient.java:47) ~[spring-data-elasticsearch-4.0.0.jar!/:4.0.0]
    at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:229) ~[elasticsearch-rest-high-level-client-6.0.1.jar!/:6.0.1]
    at org.springframework.data.elasticsearch.rest.ElasticsearchRestClient.doExecute(ElasticsearchRestClient.java:208) ~[spring-data-elasticsearch-4.0.0.jar!/:4.0.0]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.0.1.jar!/:6.0.1]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) ~[elasticsearch-6.0.1.jar!/:6.0.1]
    at org.springframework.data.elasticsearch.rest.request.SingleIndexBulkRequestBuilder.execute(SingleIndexBulkRequestBuilder.java:58) ~[spring-data-elasticsearch-4.0.0.jar!/:4.0.0]
    at org.springframework.data.elasticsearch.core.ElasticsearchTemplate.bulkIndex(ElasticsearchTemplate.java:596) ~[spring-data-elasticsearch-4.0.0.jar!/:4.0.0]
    at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.saveAll(AbstractElasticsearchRepository.java:184) ~[spring-data-elasticsearch-4.0.0.jar!/:4.0.0]
    at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:377) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:629) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:593) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61) ~[spring-data-commons-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at com.sun.proxy.$Proxy100.saveAll(Unknown Source) ~[na:na]
    at io.kaicode.elasticvc.api.VersionControlHelper.endOldVersions(VersionControlHelper.java:237) ~[elasticvc-2.0.1.jar!/:na]
    at io.kaicode.elasticvc.api.ComponentService.doSaveBatchComponents(ComponentService.java:74) ~[elasticvc-2.0.1.jar!/:na]
    at org.snomed.snowstorm.core.data.services.ReferenceSetMemberService.doSaveBatchMembers(ReferenceSetMemberService.java:210) ~[classes!/:2.2.2]
    at org.snomed.snowstorm.core.data.services.ConceptService.doSaveBatchComponents(ConceptService.java:836) ~[classes!/:2.2.2]
    at org.snomed.snowstorm.core.data.services.ConceptService$$FastClassBySpringCGLIB$$aae2a414.invoke(<generated>) ~[classes!/:2.2.2]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:685) ~[spring-aop-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.snomed.snowstorm.core.data.services.ConceptService$$EnhancerBySpringCGLIB$$40075d1a.doSaveBatchComponents(<generated>) ~[classes!/:2.2.2]
    at org.snomed.snowstorm.core.data.services.ReleaseService.releaseComponentsOfType(ReleaseService.java:71) ~[classes!/:2.2.2]
    at org.snomed.snowstorm.core.data.services.ReleaseService.createVersion(ReleaseService.java:47) ~[classes!/:2.2.2]
    at org.snomed.snowstorm.core.data.services.CodeSystemService.createVersion(CodeSystemService.java:106) ~[classes!/:2.2.2]
    at org.snomed.snowstorm.rest.CodeSystemController.createVersion(CodeSystemController.java:60) ~[classes!/:2.2.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) ~[spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) ~[spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) [spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) [spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) [spring-webmvc-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at io.kaicode.rest.util.branchpathrewrite.BranchPathUriRewriteFilter.doFilter(BranchPathUriRewriteFilter.java:50) [branch-path-uri-rewrite-filter-2.0.0.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.snomed.snowstorm.rest.security.RequestHeaderAuthenticationDecoratorWithRequiredRole.doFilterInternal(RequestHeaderAuthenticationDecoratorWithRequiredRole.java:43) [classes!/:2.2.2]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.5.RELEASE.jar!/:5.0.5.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.29.jar!/:8.5.29]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
    Suppressed: java.lang.IllegalStateException: Elasticsearch didn't return the [Content-Type] header, unable to parse response body
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:551) ~[elasticsearch-rest-high-level-client-6.0.1.jar!/:6.0.1]
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:534) ~[elasticsearch-rest-high-level-client-6.0.1.jar!/:6.0.1]
        ... 131 common frames omitted
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/es-member/_bulk?timeout=1m], status line [HTTP/1.1 413 Request Entity Too Large]

    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:355) ~[elasticsearch-rest-client-6.0.1.jar!/:6.0.1]
    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:344) ~[elasticsearch-rest-client-6.0.1.jar!/:6.0.1]
    at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:123) ~[httpcore-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:439) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:329) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.9.jar!/:4.4.9]
    ... 1 common frames omitted
kaicode commented 5 years ago

I downloaded a copy of the Veterinary Extension. It looks like the effectiveTime values include the time in addition to the date, it's the first time I've seen that. I'm not sure how Snowstorm will behave with effectiveTime values using this format. This is probably the reason why a version was not created automatically.

Your stack trace does seem to show a bug in Snowstorm which occurs when updating a very large number components at once. It's possible that the version date you provided is being applied to every version in your extension because it does not recognise the existing timestamps. Most operations in Snowstorm happen in batches but it looks like the versioning process needs bringing in to line.

We will get this versioning bug fixed for the next public release which will be a version 3.x release currently planned for the beginning of May.

kaicode commented 5 years ago

Just adding a further comment to this. I've been in touch to confirm with the SNOMED Education and Implementation team and a timestamp in the effectiveTime field is not supported for released RF2 content. Snowstorm only supports the date format YYYYMMDD for the effectiveTime field in RF2.

dkincaid commented 5 years ago

That is interesting. Thanks for following up. I'll contact the maintainers of the veterinary extension and let them know. Hopefully I can convince them to change it. In the meantime I will create a script locally that will modify that field and rewrite the RF2 file so that it complies with the standard. I appreciate your time looking into it.

dkincaid commented 5 years ago

You can close this issue if you want. I'll fix the RF2 files for the extension and try importing into Snowstorm again. I'll let you know how it goes, but I probably won't get a chance to try it until later this week.

greenvet2 commented 5 years ago

We utilize timestamp for our effectiveTime field because our work for USDA requires the ability to have immediate addition and recall of content for outbreak situations (they call our server each day to get updates from the last 24 hours). I am surprised that the SNOMED Education and Implementation team would say it's not supported since the format for time inclusion is specifically given in the Release File Specification documentation in the IHTSDO Document Library. (https://confluence.ihtsdotools.org/display/DOCRELFMT/3.1.2+Release+File+Data+Types) I know that the International release and other release centers publish as a "releases" with only date, but the specification does clearly state that when time is needed (as in our situation) the format for the timestamp is YYYYMMDDThhmmssZ, which is what we have used.

kaicode commented 5 years ago

@greenvet2 Thank you for getting in touch and expanding on your use case.

You are right that the Time data type within the Release File Specification includes both formats, however the specs for each of the component file formats (concept, description, relationship and reference set) expand on this definition and state: Note: In distribution files the effectiveTime should follow the short ISO date format (YYYYMMDD) and should not include the hours, minutes, seconds or timezone indicator.

For further clarification the product support team can be contacted at eps@snomed.org.

Kind regards, Kai