changes for improved kitodo docker compose

[markusweigelt]

Branch includes a new structuring of the sample data and an implementation to make them available in Kitodo.Production now.

[bertsky]

I need https://github.com/markusweigelt/kitodo-production-docker/pull/10 to start up without errors, but still when I enter the App URL I end up on http://localhost:8080/kitodo/pages/login.jsf – a nearly empty page without login input fields.

[bertsky]

With the current version I get a new problem when issueing the OCR task:

result page with stack trace

``` java.lang.IllegalArgumentException: /usr/local/kitodo/rulesets/ruleset_default.xml (No such file or directory) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:196) at org.kitodo.utils.JAXBContextCache.getUnmarshalled(JAXBContextCache.java:90) at org.kitodo.dataeditor.ruleset.RulesetManagement.read(RulesetManagement.java:283) at org.kitodo.dataeditor.ruleset.RulesetManagement.load(RulesetManagement.java:221) at org.kitodo.production.helper.metadata.legacytypeimplementations.LegacyPrefsHelper.loadPrefs(LegacyPrefsHelper.java:130) at org.kitodo.production.services.data.RulesetService.getPreferences(RulesetService.java:210) at org.kitodo.production.services.data.TaskService.executeScript(TaskService.java:476) at org.kitodo.production.services.command.KitodoScriptService.runScript(KitodoScriptService.java:453) at org.kitodo.production.services.command.KitodoScriptService.executeOtherScript(KitodoScriptService.java:182) at org.kitodo.production.services.command.KitodoScriptService.executeScript(KitodoScriptService.java:166) at org.kitodo.production.services.command.KitodoScriptService.execute(KitodoScriptService.java:121) at org.kitodo.production.forms.ProcessForm.executeKitodoScriptForProcesses(ProcessForm.java:645) at org.kitodo.production.forms.ProcessForm.executeKitodoScriptSelection(ProcessForm.java:639) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.el.parser.AstValue.invoke(AstValue.java:252) at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:266) at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40) at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50) at org.jboss.weld.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40) at org.jboss.weld.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50) at org.apache.myfaces.view.facelets.el.ContextAwareTagMethodExpression.invoke(ContextAwareTagMethodExpression.java:96) at org.apache.myfaces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:74) at org.springframework.faces.webflow.FlowActionListener.processAction(FlowActionListener.java:71) at org.springframework.faces.model.SelectionTrackingActionListener.processAction(SelectionTrackingActionListener.java:64) at javax.faces.component.UICommand.broadcast(UICommand.java:120) at javax.faces.component.UIViewRoot._broadcastAll(UIViewRoot.java:1255) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:420) at javax.faces.component.UIViewRoot._process(UIViewRoot.java:1741) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:935) at org.apache.myfaces.lifecycle.InvokeApplicationExecutor.execute(InvokeApplicationExecutor.java:42) at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:195) at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:142) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:204) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.kitodo.production.servletfilter.EncodingFilter.doFilter(EncodingFilter.java:69) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.omnifaces.filter.GzipResponseFilter.doFilter(GzipResponseFilter.java:181) at org.omnifaces.filter.HttpFilter.doFilter(HttpFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327) at org.kitodo.production.security.SecurityObjectAccessFilter.doFilter(SecurityObjectAccessFilter.java:86) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:147) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Unknown Source) [Impressum](http://sdvocr:8080/kitodo/pages/processes.jsf?tabIndex=0#) [Datenschutz](http://sdvocr:8080/kitodo/pages/processes.jsf?tabIndex=0#) [Nutzungsbedingungen](http://sdvocr:8080/kitodo/pages/processes.jsf?tabIndex=0#) KITODO.PRODUCTION Version 3.4.3-SNAPSHOT ```

[markusweigelt]

@bertsky please update branch an subrepos there are some updates regarding that problem

[markusweigelt]

@bertsky please update again. i add the build config to manager docker-compose so it's necessary to run fresh build.

In my case following error occurs after ocrd-import.

result page with stack trace

```Jun 17 14:50:08 ocrd-manager for_production.sh: running with 3 26 /data/3 deu Fraktur CONTROLLER=ocrd-controller:22 Jun 17 14:50:08 ocrd-manager for_production.sh: ocr_exit in async mode - immediate termination of the script Jun 17 14:50:08 ocrd-manager for_production.sh: '/data/3/images' -> 'ocr-d//data/3' Jun 17 14:50:08 ocrd-manager for_production.sh: '/data/3/images/FILE_0014_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0014_ORIGINAL.jpg' Jun 17 14:50:08 ocrd-manager for_production.sh: '/data/3/images/FILE_0013_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0013_ORIGINAL.jpg' Jun 17 14:50:08 ocrd-manager for_production.sh: '/data/3/images/FILE_0011_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0011_ORIGINAL.jpg' Jun 17 14:50:08 ocrd-manager for_production.sh: '/data/3/images/FILE_0012_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0012_ORIGINAL.jpg' Jun 17 14:50:08 ocrd-manager for_production.sh: '/data/3/images/FILE_0010_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0010_ORIGINAL.jpg' Jun 17 14:50:08 ocrd-manager for_production.sh: execute commands via ssh by the controller Jun 17 14:50:13 ocrd-manager for_production.sh: /data/ocr-d/data/3 Jun 17 14:50:38 ocrd-manager for_production.sh: Traceback (most recent call last): Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/bin/ocrd", line 33, in Jun 17 14:50:38 ocrd-manager for_production.sh: sys.exit(load_entry_point('ocrd', 'console_scripts', 'ocrd')()) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1128, in __call__ Jun 17 14:50:38 ocrd-manager for_production.sh: return self.main(*args, **kwargs) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1053, in main Jun 17 14:50:38 ocrd-manager for_production.sh: rv = self.invoke(ctx) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1659, in invoke Jun 17 14:50:38 ocrd-manager for_production.sh: return _process_result(sub_ctx.command.invoke(sub_ctx)) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1659, in invoke Jun 17 14:50:38 ocrd-manager for_production.sh: return _process_result(sub_ctx.command.invoke(sub_ctx)) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1395, in invoke Jun 17 14:50:38 ocrd-manager for_production.sh: return ctx.invoke(self.callback, **ctx.params) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 754, in invoke Jun 17 14:50:38 ocrd-manager for_production.sh: return __callback(*args, **kwargs) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/build/core/ocrd/ocrd/cli/validate.py", line 112, in validate_process Jun 17 14:50:38 ocrd-manager for_production.sh: _inform_of_result(validate_tasks([ProcessorTask.parse(t) for t in tasks], Workspace(Resolver(), directory=workspace), page_id=page_id, overwrite=overwrite)) Jun 17 14:50:38 ocrd-manager for_production.sh: File "/build/core/ocrd/ocrd/task_sequence.py", line 169, in validate_tasks Jun 17 14:50:38 ocrd-manager for_production.sh: task.validate() Jun 17 14:50:38 ocrd-manager for_production.sh: File "/build/core/ocrd/ocrd/task_sequence.py", line 71, in validate Jun 17 14:50:38 ocrd-manager for_production.sh: raise Exception("Executable not found in PATH: %s" % self.executable) Jun 17 14:50:38 ocrd-manager for_production.sh: Exception: Executable not found in PATH: ocrd-ocrd```

[bertsky]

please update branch an subrepos there are some updates regarding that problem

I still get the error though...

In my case following error occurs after ocrd-import.

Could you please post the logs before that? (We should now see the workflow itself during init.)

[markusweigelt]

@bertsky logs before traceback

In my case following error occurs after ocrd-import.

Could you please post the logs before that? (We should now see the workflow itself during init.)

# ocrd-controller:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7

# ocrd-controller:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7

# ocrd-controller:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7

[ OK ] OpenBSD Secure Shell server sshd        

[ OK ] enhanced syslogd rsyslogd        

Jun 23 14:00:34 ocrd-manager rsyslogd: rsyslogd's groupid changed to 102

Jun 23 14:00:34 ocrd-manager rsyslogd: rsyslogd's userid changed to 101

Jun 23 14:00:34 ocrd-manager rsyslogd:  [origin software="rsyslogd" swVersion="8.32.0" x-pid="57" x-info="http://www.rsyslog.com"] start

Jun 23 14:09:06 ocrd-manager for_production.sh: ocr_init initialize variables and directory structure

Jun 23 14:09:06 ocrd-manager for_production.sh: running with 3 26 /data/3 deu Fraktur CONTROLLER=ocrd-controller:22

Jun 23 14:09:06 ocrd-manager for_production.sh: ocr_exit in async mode - immediate termination of the script

Jun 23 14:09:06 ocrd-manager for_production.sh: '/data/3/images' -> 'ocr-d//data/3'

Jun 23 14:09:06 ocrd-manager for_production.sh: '/data/3/images/FILE_0013_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0013_ORIGINAL.jpg'

Jun 23 14:09:06 ocrd-manager for_production.sh: '/data/3/images/FILE_0010_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0010_ORIGINAL.jpg'

Jun 23 14:09:06 ocrd-manager for_production.sh: '/data/3/images/FILE_0014_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0014_ORIGINAL.jpg'

Jun 23 14:09:06 ocrd-manager for_production.sh: '/data/3/images/FILE_0011_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0011_ORIGINAL.jpg'

Jun 23 14:09:06 ocrd-manager for_production.sh: '/data/3/images/FILE_0012_ORIGINAL.jpg' -> 'ocr-d//data/3/FILE_0012_ORIGINAL.jpg'

Jun 23 14:09:06 ocrd-manager for_production.sh: execute commands via ssh by the controller

Jun 23 14:09:13 ocrd-manager for_production.sh: /data/ocr-d/data/3

Jun 23 14:09:41 ocrd-manager for_production.sh: Traceback (most recent call last):

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/bin/ocrd", line 33, in <module>

Jun 23 14:09:41 ocrd-manager for_production.sh:     sys.exit(load_entry_point('ocrd', 'console_scripts', 'ocrd')())

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1128, in __call__

Jun 23 14:09:41 ocrd-manager for_production.sh:     return self.main(*args, **kwargs)

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1053, in main

Jun 23 14:09:41 ocrd-manager for_production.sh:     rv = self.invoke(ctx)

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1659, in invoke

Jun 23 14:09:41 ocrd-manager for_production.sh:     return _process_result(sub_ctx.command.invoke(sub_ctx))

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1659, in invoke

Jun 23 14:09:41 ocrd-manager for_production.sh:     return _process_result(sub_ctx.command.invoke(sub_ctx))

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1395, in invoke

Jun 23 14:09:41 ocrd-manager for_production.sh:     return ctx.invoke(self.callback, **ctx.params)

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 754, in invoke

Jun 23 14:09:41 ocrd-manager for_production.sh:     return __callback(*args, **kwargs)

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/build/core/ocrd/ocrd/cli/validate.py", line 112, in validate_process

Jun 23 14:09:41 ocrd-manager for_production.sh:     _inform_of_result(validate_tasks([ProcessorTask.parse(t) for t in tasks], Workspace(Resolver(), directory=workspace), page_id=page_id, overwrite=overwrite))

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/build/core/ocrd/ocrd/task_sequence.py", line 169, in validate_tasks

Jun 23 14:09:41 ocrd-manager for_production.sh:     task.validate()

Jun 23 14:09:41 ocrd-manager for_production.sh:   File "/build/core/ocrd/ocrd/task_sequence.py", line 71, in validate

Jun 23 14:09:41 ocrd-manager for_production.sh:     raise Exception("Executable not found in PATH: %s" % self.executable)

Jun 23 14:09:41 ocrd-manager for_production.sh: Exception: Executable not found in PATH: ocrd-ocrd

[markusweigelt]

@bertsky Sorry for the last change. I'll try to pay more attention to .env changes.

[bertsky]

If you update the Manager (after merging https://github.com/markusweigelt/ocrd_manager/pull/25) and Controller (with latest master), then everything should work again. (With logging now mirrored between Controller and Manager, and workspace names independent between Controller and Manager.)

[markusweigelt]

@bertsky Still made the changes discussed. In the docker-compose for the manager I added an environment variable MANAGER_CONTROLLER_PORT_SSH to distinguish the container port from the host port cause of conflict in managed mode.

[bertsky]

In the docker-compose for the manager I added an environment variable MANAGER_CONTROLLER_PORT_SSH to distinguish the container port from the host port cause of conflict in managed mode.

@markusweigelt I don't understand that. Where is this variable used/referenced? Why not just use CONTROLLER_PORT_SSH?

[markusweigelt]

@markusweigelt I don't understand that. Where is this variable used/referenced? Why not just use CONTROLLER_PORT_SSH?

@bertsky CONTROLLER_PORT_SSH is the port of container to the host system. In default network in bridge mode the internal container ssh port in manager is used. So to distinguish between these two ports and to maybe contect to controller with ssh from host system I have added these env.