search

daisy / pipeline-webui

A Web User Interface for the DAISY Pipeline 2
Other
3 stars 2 forks source link

Add a docker-compose file #114

Closed egli closed 7 years ago

egli commented 7 years ago

A very simple docker-compose.yml that starts a pipeline in one container and the webui in another container. Start the system with 

$ docker-compose up

Inside the webui container the pipeline is visible under http://pipeline:8181. At the moment however afaik the pipeline URL is hardcoded in the webui. Afaik @josteinaj produced a patch to fix this. So take this PR as a work in progress until the images for both the pipeline and the webui stabilize.

egli commented 7 years ago

I rebased this PR onto develop

egli commented 7 years ago

Now the environment of the webui image is set up so that it should connect to the pipeline:

eglic@wharton:~/src/pipeline-webui$ docker-compose exec webui env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=d997e8392772
TERM=xterm
DAISY_PIPELINE2_URL=http://pipeline:8181/ws
LANG=C.UTF-8
JAVA_HOME=/docker-java-home/jre
JAVA_VERSION=8u141
JAVA_DEBIAN_VERSION=8u141-b15-1~deb9u1
CA_CERTIFICATES_JAVA_VERSION=20170531+nmu1
HOME=/root
eglic@wharton:~/src/pipeline-webui$

but still the web seems to try to connect to localhost

webui_1     | org.daisy.pipeline.client.Pipeline2Exception: Error while GETing.
webui_1     |   at org.daisy.pipeline.client.http.Pipeline2HttpClient.getDelete(Pipeline2HttpClient.java:100) ~[org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]
webui_1     |   at org.daisy.pipeline.client.http.Pipeline2HttpClient.get(Pipeline2HttpClient.java:61) ~[org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]
webui_1     |   at org.daisy.pipeline.client.http.WS.alive(WS.java:100) [org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]
webui_1     |   at org.daisy.pipeline.client.http.WS.checkIfLocal(WS.java:656) [org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]
webui_1     |   at org.daisy.pipeline.client.http.WS.<init>(WS.java:57) [org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]
webui_1     |   at controllers.Application.<clinit>(Application.java:75) [org.daisy.pipeline.webui-2.5.2-SNAPSHOT-sans-externalized.jar:na]
webui_1     |   at Global$1.run(Global.java:72) [org.daisy.pipeline.webui-2.5.2-SNAPSHOT-sans-externalized.jar:na]
webui_1     |   at akka.actor.LightArrayRevolverScheduler$$anon$2$$anon$1.run(Scheduler.scala:242) [com.typesafe.akka.akka-actor_2.11-2.3.11.jar:na]
webui_1     |   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [com.typesafe.akka.akka-actor_2.11-2.3.11.jar:na]
webui_1     |   at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [com.typesafe.akka.akka-actor_2.11-2.3.11.jar:na]
webui_1     | Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8181 refused
webui_1     |   at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:158) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) ~[org.apache.httpcomponents.httpclient-4.1.3.jar:4.1.3]
webui_1     |   at org.daisy.pipeline.client.http.Pipeline2HttpClient.getDelete(Pipeline2HttpClient.java:96) ~[org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]
webui_1     |   at org.daisy.pipeline.client.http.Pipeline2HttpClient.get(Pipeline2HttpClient.java:61) ~[org.daisy.pipeline.clientlib-java-httpclient-2.0.1.jar:na]

@josteinaj any idea why this could be happening? Is the environment not passed through the activator script?

bertfrees commented 7 years ago

Does the webui call the setEndPoint method somewhere? It doesn't look like it.

Also related is this issue: https://github.com/daisy/pipeline-clientlib-java/issues/20.

egli commented 7 years ago

Even when adding 

expose:
      - "8181"

to the pipeline service I still get

eglic@wharton:~/src/pipeline-webui$ docker-compose exec pipeline wget http://localhost:8181/ws
--2017-10-02 15:58:01--  http://localhost:8181/ws
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:8181... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-10-02 15:58:02 ERROR 404: Not Found.

eglic@wharton:~/src/pipeline-webui$ docker-compose exec webui wget http://pipeline:8181/ws
--2017-10-02 15:58:08--  http://pipeline:8181/ws
Resolving pipeline (pipeline)... 172.18.0.2
Connecting to pipeline (pipeline)|172.18.0.2|:8181... failed: Connection refused.
bertfrees commented 7 years ago

Does it help if you set org.daisy.pipeline.ws.host to 0.0.0.0?

egli commented 7 years ago

@bertfrees yes, changing org.daisy.pipeline.ws.host in system.properties to 0.0.0.0 helps. I can now connect to the pipeline

$ docker run --name pipeline --rm --env-file=docker-envfile -p 8181:8181 daisyorg/pipeline:host0
eglic@wharton:~/src/pipeline-assembly$ ~/src/pipeline-cli-go/build/bin/dp2 version

Client version:                 2.0.1-SNAPSHOT         
Pipeline version:               1.10.3
Pipeline authentication:        false

I can turn auth on and off via the env vars that I define in the docker-envfile

However when I try so set the host via an env var and then setting the corresponding property in the startup script I can no longer connect to the pipeline.

I think I'll take this issue to the assembly

egli commented 7 years ago

OK, docker-compose up now brings up a more or less working system. What we should probably look at before merging this is the fact that there is no persistence, i.e. everytime you start docker-compose you will land in the first use screen.

egli commented 7 years ago

OK, IMHO the docker-compose file is now ready

  1. it adds a separate volume to deal with the persistence issue, i.e. the data that the webui stores should be kept across restarts of the image
  2. The exceptions are a problem with the webui itself and afaik @josteinaj is taking care of them
bertfrees commented 7 years ago

@egli I read somewhere that links: is deprecated. Do you know something more about that?

egli commented 7 years ago

@bertfrees I think you are supposed to define user defined networks now. But IMHO for simple use cases I think the link feature is way more straight forward.

bertfrees commented 7 years ago

Ok.

egli commented 7 years ago

So, as discussed I downgraded the docker-compose.yml file to version 2.1 so that we can make use of the healthcheck feature, where one service waits for another until it is healthy.

version: "2.1"
services:
  pipeline:
    image: daisyorg/pipeline2
    environment:
      PIPELINE2_AUTH: "false"
      PIPELINE2_LOCAL: "false"
      PIPELINE2_HOST: "pipeline"
  webui:
    image: daisyorg/webui
    environment:
      DAISY_PIPELINE2_URL: http://pipeline:8181/ws
    ports:
      - "9000:9000"
    volumes:
      - "webui-data:/opt/pipeline2-webui/data"
    depends_on:
      pipeline:
        condition: service_healthy

volumes:
  webui-data:

I can start docker-compose and it seems to wait for the pipeline2, however apparently pipeline2 never reaches the healthy state

eglic@wharton:~/src/pipeline-webui$ docker-compose up
Creating network "pipelinewebui_default" with the default driver
Creating pipelinewebui_pipeline_1 ... 
Creating pipelinewebui_pipeline_1 ... done

ERROR: for webui  Container "90f6913229d7" is unhealthy.
ERROR: Encountered errors while bringing up the project.

When looking at the logs with docker-compose logs -f it seems that the pipeline has come up successfully. And also if I bring up the pipeline2 image stand alone it reaches the healthy state quite happily.

eglic@wharton:~/src/pipeline-webui$ docker run -d --rm daisyorg/pipeline2
4c226e85737d97ff711489593f0c4b2d00c9eda23bdec89c68120acd4d626b57
eglic@wharton:~/src/pipeline-webui$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                             PORTS               NAMES
4c226e85737d        daisyorg/pipeline2   "/opt/daisy-pipeli..."   25 seconds ago      Up 23 seconds (health: starting)   8181/tcp            gifted_northcutt
eglic@wharton:~/src/pipeline-webui$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                    PORTS               NAMES
4c226e85737d        daisyorg/pipeline2   "/opt/daisy-pipeli..."   49 seconds ago      Up 47 seconds (healthy)   8181/tcp            gifted_northcutt

So, tbh, since this feature was only available such a short time maybe it never quite worked right. I'm not so sure if we should really invest more time into this.

bertfrees commented 7 years ago

What if you pass the same environment variables when starting it stand alone?

egli commented 7 years ago

It comes up without a problem:

eglic@wharton:~/src/pipeline-webui$ docker run -d --rm --env PIPELINE2_AUTH=false --env PIPELINE2_LOCAL=false daisyorg/pipeline2
ea4c9644b2e06054cef65eb0221c89dd477d2040ea976a503b63aa469a748a02
eglic@wharton:~/src/pipeline-webui$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                            PORTS               NAMES
ea4c9644b2e0        daisyorg/pipeline2   "/opt/daisy-pipeli..."   6 seconds ago       Up 3 seconds (health: starting)   8181/tcp            eager_feynman
eglic@wharton:~/src/pipeline-webui$ docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                    PORTS               NAMES
ea4c9644b2e0        daisyorg/pipeline2   "/opt/daisy-pipeli..."   33 seconds ago      Up 31 seconds (healthy)   8181/tcp            eager_feynman

Obvously I cannot set the PIPELINE2_HOST. If I do that it complains that it cannot resolve the host. Which makes sense since the network is missing if use it outside of docker compose.

bertfrees commented 7 years ago

Ah yes, of course.

I have a feeling it has to do with PIPELINE2_HOST. The health check should probably become:

HEALTHCHECK --interval=30s --timeout=10s --start-period=1m CMD curl --fail http://${PIPELINE2_HOST-localhost}:8181/ws/alive || exit 1
bertfrees commented 7 years ago

And after my environment variables change it should be:

HEALTHCHECK --interval=30s --timeout=10s --start-period=1m CMD curl --fail http://${PIPELINE2_WS_HOST-localhost}:${PIPELINE2_WS_PORT-8181}/${PIPELINE2_WS_PATH-ws}/alive || exit 1
egli commented 7 years ago

OK, I pushed a change to both the pipeline Dockerimage and to the docker-compose file in the webui. It now works as expected

bertfrees commented 7 years ago

Depends on https://github.com/daisy/pipeline-webui/pull/113

bertfrees commented 7 years ago

The docker-compose file is basically there to help you launch a server instance and a web UI instance and let them talk to each other. The Dockerfile (other PR) is there to build the image with the web UI in it.

rdeltour commented 7 years ago

The docker-compose file is basically there to help you launch a server instance and a web UI instance and let them talk to each other. The Dockerfile (other PR) is there to build the image with the web UI in it.

Yes, that I gathered. But the two PRs have some common commits, then diverge, so wasn't sure how these articulate, if some merging conflicts were to be expected.

bertfrees commented 7 years ago

Yeah I don't understand why Christian made two PR's. 114 depends on 113 so they need to be merged. I thought that was already done but maybe I only did it locally.

egli commented 7 years ago

I thought it would be better to keep the two things separate. The Docker image is useful in itself without having to use the docker-compose command. Docker compose brings the pipeline and the webui images together (and exposed its own set of problems). So while the one depends on the other I thought the two were different enough to warrant a separate PR.

josteinaj commented 7 years ago

I can rebase before merging, that's no problem. There shouldn't be any conflicts.

So I can merge #113 and #114 into master now?

bertfrees commented 7 years ago

Yes! Thanks.

josteinaj commented 7 years ago

Closed by #118.