Documentation for deploying Sync Endpoint without Docker

[chlarsen]

Dear All, As much as I appreciate the longitudinal capabilities of ODK-X, I started to get a major headache with its sync endpoint ("server") setup. The latter is purely docker-based, throwing together Java, PostgreSQL and OpenLDAP components for "easy" deployment. Alas, this is not, how it works in my FreeBSD world, where we have jails, which contain carefully and securely crafted server instances, e.g. one jail for OpenLDAP, another one for PostGreSQL, another one for mail service, etc. As a result, the ODK-X endpoint sync facility could quite simply fit into a well-prepared bed, sitting inside its own tomcat jail, and all is fine, easy (compartimentalised) to maintain and still transparent. Docker, on the contrary, installs a lot of things in my Debian VM sitting on top of FreeBSD, and forces me to run a second PostgreSQL and OpenLDAP server - no way! Plus, die prescribed installation process described in the documentation just does not work, and times out here when running mvi clean install:

WARNING: An internal object pool swallowed an Exception.
org.apache.commons.dbcp2.LifetimeExceededException: The lifetime of the connection [590,066] milliseconds exceeds the maximum permitted value of [590,000] milliseconds
    at org.apache.commons.dbcp2.PoolableConnectionFactory.validateLifetime(PoolableConnectionFactory.java:637)
    at org.apache.commons.dbcp2.PoolableConnectionFactory.passivateObject(PoolableConnectionFactory.java:417)
    at org.apache.commons.pool2.impl.GenericObjectPool.returnObject(GenericObjectPool.java:550)
    at org.apache.commons.dbcp2.PoolableConnection.close(PoolableConnection.java:200)
    at org.apache.commons.dbcp2.DelegatingConnection.closeInternal(DelegatingConnection.java:239)
    at org.apache.commons.dbcp2.DelegatingConnection.close(DelegatingConnection.java:210)
    at org.apache.commons.dbcp2.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:247)
    at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:348)
    at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:335)
    at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:302)
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:661)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:875)
    at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:936)
    at org.opendatakit.common.persistence.engine.pgres.DatastoreImpl.putEntity(DatastoreImpl.java:956)
    at org.opendatakit.common.persistence.QueryResultTestIT$DataValue.assertInstance(QueryResultTestIT.java:123)
    at org.opendatakit.common.persistence.QueryResultTestIT.oneTimeSetUp(QueryResultTestIT.java:61)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Therefore, I feel compelled, for the sake of increased transparency, and a more resource-efficient approach that can take advantage of existing infrastructure, can you please, please, please provide documentation on how to install the ODK-X sync endpoint ("server') instance, only, addressing the Java side of things, without involving docker? I would be extremely grateful and you would help a lot of surveillance folks out there in the field, trying to run their own server in easy-to-understand and transparent ways. A huge thank you in advance! Chris

[Anushka-shukla]

Hey, Can I work on this issue?

[wbrunette]

@Anushka-shukla thanks you for working on this issue.

After reviewing this issue I am not sure it's an easy win, so I removed the easy win tag. Do you have experience with setting up a Java web server (e.g., Apache Tomcat) and deploying WAR files? Basically to solve this issue you will need to document how to setup their Java web server instance without using our automated scripts (basically outline the steps) without Docker. To understand the build process you will want to check out the Github Sync-Endpoint repository. You can compare two building processes for Sync-Endpoint one we use for testing and one we use for the Docker Swarm. The two modules are 'sync-endpoint-docker-test' module and the 'sync-endpoint-docker-swarm' module. You will also probably need to document the properties files to access the databases. Examples of the properties files are in 'mysql-test' and 'postgres-test'.

I have assigned the issue to you for now. Let me know if you want to try a different issue instead and I will remove the assignment.

[chlarsen]

Dear All, Thank you so much for that. I am presently swamped with - don't laugh - conventional ODK work, but will very much appreciate the opening of the docker and swarm box. I am quite familiar with the processes mentioned above, so chances are, we can get this sorted. A huge "thank you"!

[Anushka-shukla]

@wbrunette I'm not sure If I can do this but I want to give it a try by doing some research. Can you list out few resources if possible?

[wbrunette]

Here are instructions for setting it up with Docker: https://docs.odk-x.org/sync-endpoint-manual-setup/

We are abstracting the setup of Tomcat 8.5, the LDAP server, and the Postgres database.

It will probably be helpful to start with setting up a tomcat-8.5 instance and creating a "Hello World" style application to understand the Tomcat WAR setup so you can apply that to setting up the Sync-Endpoint WAR. https://tomcat.apache.org/tomcat-8.5-doc/index.html

@chlarsen please feel free to provide pointers and advice.

[Anushka-shukla]

@wbrunette I'm not able to initialise swarm.

Error: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/swarm/init: dial unix /var/run/docker.sock: connect: permission denied

Tried searching on StackOverflow but couldn't find any solution. Can you please suggest something?

[Anushka-shukla]

It's solved! I was not giving the ip address of the manger -- "docker swarm init --advertise-addr 192.168.99.100:2376 "

[CtripleU]

Hello, @chlarsen.

I'd like to resolve this. Could you please assign it to me? Thank you.

[wbrunette]

@CtripleU I have assigned the issue to you.

[CtripleU]

Thank you @wbrunette!

[CtripleU]

Hi, @wbrunette. Since there is no page in the documentation for this, can I create one?

[elmps2018]

Hi @CtripleU! Good idea to create a new page for this. Please also investigate and suggest where this should fall in relation to other pages in the TOC.

[CtripleU]

Alright, thank you for the clarification @elmps2018.

[CtripleU]

Hello, @elmps2018. I was trying to deploy sync endpoint using docker first following the documentation, but mvn clean install keeps throwing an error -

Failed to execute goal io.fabric8:docker-maven-plugin:0.34.1:start (start) on project postgres-test: I/O Error: Unable to create network for [odk-network] : {"message":"Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i br-8ea8f9c4b9f4 -j RETURN: iptables: No chain/target/match by that name.\n (exit status 1))"} (Internal Server Error: 500) -> [Help 1]

Could you please assist with this? Thank you. I have attached a screenshot.

[wbrunette]

Is Docker running? What processes are currently running in docker?

From: Cumi Oyemike @.> Sent: Wednesday, October 20, 2021 11:23 AM To: odk-x/tool-suite-X @.> Cc: wbrunette @.>; Mention @.> Subject: Re: [odk-x/tool-suite-X] Documentation for deploying Sync Endpoint without Docker (#180)

Hello, @elmps2018https://github.com/elmps2018. I was trying to deploy sync endpoint using docker first following the documentation, but mvn clean install keeps throwing an error -

Failed to execute goal io.fabric8:docker-maven-plugin:0.34.1:start (start) on project postgres-test: I/O Error: Unable to create network for [odk-network] : {"message":"Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i br-8ea8f9c4b9f4 -j RETURN: iptables: No chain/target/match by that name.\n (exit status 1))"} (Internal Server Error: 500) -> [Help 1]

Could you please assist with this? Thank you. I have attached a screenshot. [Screenshot from 2021-10-20 19-21-30]https://user-images.githubusercontent.com/34113547/138149497-8d424888-9a9a-456e-b28e-93c1de9c60a2.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/odk-x/tool-suite-X/issues/180#issuecomment-947923759, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2XP2YRMCE4D3B5X3DUC33UH4CG7ANCNFSM4WI6ZMNA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[elmps2018]

@CtripleU, my docker skills are limited but hopefully @wbrunette can help!

[CtripleU]

Hello, @wbrunette. Yes, Docker is running. But let me try restarting it. Thank you.

[CtripleU]

Running sudo systemctl restart docker before mvn clean install worked @wbrunette. Thank you!

[CtripleU]

Hello, @wbrunette! I have been following the readme here in an attempt to set up the Sync Endpoint without Docker and have successfully moved through steps 1 to 12. I am now stuck at step 13 since there is no /libs folder under src/main where the build.xml file is supposed to be located. Could you please give any pointers that can help with this?

Thank you!

[wbrunette]

@CtripleU things have changed significantly from those directions.

The key is that Apache Tomcat takes a war (which is being generated by our maven modules). Once you have the WAR you need to get the properties file correct for Apache Tomcat to work.

You can delete steps 13 and on because the configuration has completely changed so the testing is in docker and the last 13 steps are for running the tests which is outside what is needed to complete this task

[CtripleU]

Thank you for the explanation @wbrunette . I'm still a bit confused though. Please, could you give any pointers on how the WAR can be gotten?

[wbrunette]

The module in the sync-endpoint maven project titled "sync-endpoint-war" builds the war.

[CtripleU]

Alright, thank you!