jorgejesus commented 3 years ago

Background and Motivation

For Re3gistry deployment I can only use Openshift/k8s, and therefore all systems are running on docker containers

Proposed change

Change re3gistry2 configuration procedures init-config.sh to use env variables instead of init.properties this will make docker implementation easier (or use templates with env variables)
Automatic check if database needs to implemented (run registry2_drop-and-create-and-init.sql automatically if necessary, and with env variables.
Create docker-compose as system example
Write documentation on solr instance implementation and how we can test it solr is OK, and working for re3gistry2
Reimplement webapp (if necessary) only on tomcat instead of apache2 (drop apache2 or just use one system)
Documentation on what exactly the install wizard does, and how we can configure an re3gistry instance that doesn't run install at start, just parameters that it changes.

Alternatives

Current deployment is the only alternative, and is not cloud/docker/k8s oriented

Funding

No

Additional information

Have a working instance working on openshift3.6 no major issues or problems

archaeogeek commented 3 years ago

@jorgejesus I'd be interested in contributing to this work. Do you have any existing docker files that you'd be happy to share?

jorgejesus commented 3 years ago

@archaeogeek I have docker images and content written for deployment in openshift, this is in a private repository. I will look into putting it on github.

oscar9 commented 2 years ago

Actual approach:

Go to re3gistry\dist\customize-interface\example-profile-developer-docker
Edit with text editor the .bat file and add your paths similar to the following example:

set "copyFROM=C:\Workspace\master-branch\re3gistry\dist\customize-interface\example-profile-developer-docker\" set "copyTOsourceGithub=C:\Workspace\master-branch\re3gistry\sources\" set "copyTOdistGithub=C:\Workspace\master-branch\re3gistry\dist\"

Execute .bat file. It will deploy the config files in your code
Compile Re3gistry2-builder-helper
In a terminal opened in the folder Re3gistry2-builder-helper you will find a docker-compose.yml
Execute: docker-compose up -d
It will initiate a deployment environment in docker and a database example will be also loaded
Login with name@example.com password
Following urls should be working:
http://host.docker.internal:8080/re3gistry2restapi/items/any?uri=http://host.docker.internal/registry&lang=en&format=jsonc
http://host.docker.internal/registry/rest?uri=http://host.docker.internal/registry/examplecodelist&lang=en&format=jsonc
http://host.docker.internal:8080/re3gistry2/login
http://host.docker.internal/registry
http://host.docker.internal/registry/examplecodelist

MartinTuchyna commented 2 years ago

Hi,

when do you expect to have docker version available for deployment ?

Thanks,

Martin

oscar9 commented 2 years ago

Hi @MartinTuchyna ,

We are getting ready a docker image with env variables containing the war files. We are fixing some issues in the script and we hope to have it ready in the next following week.

Best regards, Oscar

MartinTuchyna commented 2 years ago

Thanks Oscar,

please, let us know, when it will be available.

Best regards, Martin

oscar9 commented 2 years ago

Hi @MartinTuchyna ,

We have a pre-release version v.2.3.2. It would be nice if you could test it the app. We are also testing the Dockerimage but we need someone else to test it and get some feedback. We are not sure it is working fully with the new version app, also we are going to do some test with the last version also. I attached the needed files, you should put them in the main folder of the repository.

I sent it first as gist as we want first a bit of feedback from users and check if it's properly working. If needed, a branch for this issue will be created. https://gist.github.com/oscar9/3687e922dc33a369cbc08bad926339eb

Docker compose is creating a example database from ./dist/customize-interface/example-profile-developer-docker/, if you let this dump activate you should put a empty filed called system.installed inside dist/app/re3gistry2/WEB-INF/classes/configurations_files/

Also, just to remember, that if you want to compile your own war file a solution similar to the one in the docker-customize-profile could be an example. https://github.com/ec-jrc/re3gistry/tree/master/dist/customize-interface/example-profile-developer-docker/Re3gistry2-build-helper

Any comment would be appreciated. Best regards, Óscar

MartinTuchyna commented 2 years ago

Hi Oskar @oscar9 ,

thanks for an update.

Colleague of mine tried to deoploy Docker instance, but has end up with missing following files:

./dist/webapp/httpd.conf a ./dist/webapp/httpd-vhosts.conf.

Best regards,

Martin

oscar9 commented 2 years ago

Hi @MartinTuchyna ,

I updated the gist. As a first attempt, you could even deactivate the "apache" group (the frontend) inside the docker-compose file. It would be a good start to check first the API and the backend.

https://gist.github.com/oscar9/3687e922dc33a369cbc08bad926339eb

Thanks for your information. I'll keep you updated with any discoveries.

Best regards, Óscar

MartinTuchyna commented 1 year ago

Thanks @oscar9

Seems, colleague of mine managed to deploy docker instance>

https://registry.stage.geocloud.sk/re3gistry2

but what shall be the url, where we could try to login to the backend?

Thanks, Martin

MartinTuchyna commented 1 year ago

After re-fixing some port setting, seems link is able to call Tomcat. But, Tomcat is reporting some issues - see attached log. According docker-compose nothing is connecting to that container from outside, so it seems to indicate some issues inside. Could you please check it? Best regards, Martin tomcat.log

oscar9 commented 1 year ago

Hi @MartinTuchyna ,

There was a problem in the last version with the installation script. This script was creating some errors in the docker image. If I remember correctly, this is exactly the error that was happening.

Error is related to a non-configured parameter. Docker uses a persistence.xml.orig as a base file to create the persistence.xml that contains a few lines that it shouldn't have.

https://github.com/ec-jrc/re3gistry/blob/master/dist/app/re3gistry2restapi/WEB-INF/classes/META-INF/persistence.xml

You could go manually in the docker container and delete the lines.

<property name="eclipselink.logging.level" value="CONFIG"/>
<property name="eclipselink.logging.level.sql" value="CONFIG"/>
<property name="eclipselink.logging.parameters" value="false"/>
<property name="eclipselink.logging.file" value="path/to/catalina/base/logs/eclipselink.log"/>

The folder inside the tomcat container should be: /usr/local/tomcat/webapps/re3gistry2restapi/WEB-INF/classes/META-INF/persistence.xml

It should look similar to this one: https://github.com/ec-jrc/re3gistry/blob/develop/dist/app/re3gistry2restapi/WEB-INF/classes/META-INF/persistence.xml.orig

I made some progress with the current version that is in the develop branch, and I will update you as soon as possible with an updated Docker Image.

Best regards, Óscar

PetoP commented 1 year ago

Hi @oscar9,

I am colleague of @MartinTuchyna and I am responsible for our deployment of Re3gistry. Thank you for your previous message about editing persistence.xml file as it helped and Re3gistry is working after this modification (I have modified it on local copy of GIT repo and rebuilt docker image, so that change will be permanent).

However, I am struggling with e-mail setup. I have e-mail account dedicated for sending messages from Re3gistry. It has SMTP access and I have successfully tested sending mail thru it via msmtp. However I am unable to get Re3gistry to send e-mail. I have set SMTP connection parameters in all configuration files that are used in reg-tomcat image (which I have listed from Dockerfile) and also changed smtphost variable in docker-compose.yml and rebuilt reg-tomcat image, but no e-mail is coming from Re3gistry when I create new user. What is worse, there is even no error message in docker-compose logs, so I am unable to identify what is wrong with my setup.

Can you please explain me what configuration parameters I have to change in which files in order to connect reg-tomcat to desired SMTP server?

Thank you very much, Peter

MartinTuchyna commented 1 year ago

Hi @oscar9

any idea for guidance @PetoP https://github.com/ec-jrc/re3gistry/issues/22#issuecomment-1216663908

to be able to access backend https://registry.stage.geocloud.sk/re3gistry2/ ?

Thanks, Martin

oscar9 commented 1 year ago

Hi @MartinTuchyna @PetoP ,

I'm checking the email configuration in the docker image. A quick response is that maybe not all needed parameters are being configured through the script.

@MartinTuchyna not sure about your question, the login of the backend needs to be configured during the installation process. Did you installed the software? (Installation wizard)

Best regards

PetoP commented 1 year ago

Hi @oscar9,

Martin was only asking about progress with our email problem. Meanwhile, I have found SMTP error messages in re3gistry.log file caused by new user creation.

I have applied working msmtp configuration to all config files in repo containing SMTP configuration, rebuilt web image, wiped PostgreSQL database and started with clean installation each time trying different config.

In case you are unable to replicate with SMTP server you are using, I am more than willing to share (via email) credentials and SMTP configuration I am using.

Best regards, Peter

[ERROR] 2022-09-03 05:49:53.813 [http-apr-8080-exec-6] [utility.MailManager.sendMail:156] - Could not convert socket to TLS javax.mail.MessagingException: Could not convert socket to TLS at com.sun.mail.smtp.SMTPTransport.startTLS(SMTPTransport.java:1907) ~[mail-1.4.7.jar:1.4.7] at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:666) ~[mail-1.4.7.jar:1.4.7] at javax.mail.Service.connect(Service.java:317) ~[mail-1.4.7.jar:1.4.7] at javax.mail.Service.connect(Service.java:176) ~[mail-1.4.7.jar:1.4.7] at javax.mail.Service.connect(Service.java:125) ~[mail-1.4.7.jar:1.4.7] at javax.mail.Transport.send0(Transport.java:194) ~[mail-1.4.7.jar:1.4.7] at javax.mail.Transport.send(Transport.java:124) ~[mail-1.4.7.jar:1.4.7] at eu.europa.ec.re3gistry2.base.utility.MailManager.sendMail(MailManager.java:153) [Re3gistry2Base-1.0.jar:?] at eu.europa.ec.re3gistry2.base.utility.MailManager.sendMail(MailManager.java:46) [Re3gistry2Base-1.0.jar:?] at eu.europa.ec.re3gistry2.web.controller.RegistryManagerUsersAdd.processRequest(RegistryManagerUsersAdd.java:211) [classes/:?] at eu.europa.ec.re3gistry2.web.controller.RegistryManagerUsersAdd.doPost(RegistryManagerUsersAdd.java:280) [classes/:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) [servlet-api.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) [servlet-api.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at eu.europa.ec.re3gistry2.web.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:47) [classes/:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.6.0.jar:1.6.0] at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.6.0.jar:1.6.0] at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) [shiro-core-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.6.0.jar:1.6.0] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at eu.europa.ec.re3gistry2.web.controller.ValidateSalt.doFilter(ValidateSalt.java:67) [classes/:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at eu.europa.ec.re3gistry2.web.controller.LoadSalt.doFilter(LoadSalt.java:78) [classes/:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) [catalina.jar:7.0.109] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) [catalina.jar:7.0.109] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) [catalina.jar:7.0.109] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165) [catalina.jar:7.0.109] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) [catalina.jar:7.0.109] at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) [catalina.jar:7.0.109] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [catalina.jar:7.0.109] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:451) [catalina.jar:7.0.109] at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1201) [tomcat-coyote.jar:7.0.109] at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654) [tomcat-coyote.jar:7.0.109] at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2468) [tomcat-coyote.jar:7.0.109] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-coyote.jar:7.0.109] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) at sun.security.ssl.HandshakeContext.(HandshakeContext.java:171) ~[?:1.8.0_292] at sun.security.ssl.ClientHandshakeContext.(ClientHandshakeContext.java:98) ~[?:1.8.0_292] at sun.security.ssl.TransportContext.kickstart(TransportContext.java:220) ~[?:1.8.0_292] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:428) ~[?:1.8.0_292] at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:549) ~[mail-1.4.7.jar:1.4.7] at com.sun.mail.util.SocketFetcher.startTLS(SocketFetcher.java:486) ~[mail-1.4.7.jar:1.4.7] at com.sun.mail.smtp.SMTPTransport.startTLS(SMTPTransport.java:1902) ~[mail-1.4.7.jar:1.4.7] ... 59 more [ERROR] 2022-09-03 05:49:53.816 [http-apr-8080-exec-6] [controller.RegistryManagerUsersAdd.processRequest:239] - Could not convert socket to TLS javax.mail.MessagingException: Could not convert socket to TLS at eu.europa.ec.re3gistry2.base.utility.MailManager.sendMail(MailManager.java:157) ~[Re3gistry2Base-1.0.jar:?] at eu.europa.ec.re3gistry2.base.utility.MailManager.sendMail(MailManager.java:46) ~[Re3gistry2Base-1.0.jar:?] at eu.europa.ec.re3gistry2.web.controller.RegistryManagerUsersAdd.processRequest(RegistryManagerUsersAdd.java:211) [classes/:?] at eu.europa.ec.re3gistry2.web.controller.RegistryManagerUsersAdd.doPost(RegistryManagerUsersAdd.java:280) [classes/:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) [servlet-api.jar:?] at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) [servlet-api.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat7-websocket.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at eu.europa.ec.re3gistry2.web.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:47) [classes/:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [catalina.jar:7.0.109] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [catalina.jar:7.0.109] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.6.0.jar:1.6.0] at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.6.0.jar:1.6.0] at org.apache.shiro.subject.support.SubjectCallable.call

oscar9 commented 1 year ago

Hi @PetoP ,

Thanks for waiting. I think you arrive at the same problem that in issue #137 where we were waiting for an answer from the user and we didn't receive it.

I'm going to talk about this with my colleagues and I let you know as soon as possible. Probably we will ask you for the credentials if it is still not a problem, so we could replicate the error.

MartinTuchyna commented 1 year ago

Hi Oscar, any update on this issue as we can only use super admin account to access the system. Thanks, Martin and Peter

oscar9 commented 1 year ago

Hi @MartinTuchyna ,

Thank you for your patience. We were in the process of creating the new release and we had some delays. We get back to it, we will keep you informed.

MartinTuchyna commented 1 year ago

Dear @oruscalleda

any update on this topic?

ec-jrc / re3gistry

make Re3gistry more docker/k8s friendly #22

Background and Motivation

Proposed change

Alternatives

Funding

Additional information