search

simonsobs / sisock

Sisock ('saɪsɒk): streaming of Simons Obs. data over websockets for quicklook
Other
2 stars 0 forks source link

Use Docker Compose to build and run the containers #12

Closed BrianJKoopman closed 5 years ago

BrianJKoopman commented 5 years ago

As the number of DataNodeServers grows, so does the number of Docker containers. Docker Compose is a tool for defining and running multi-container applications. This commit configures docker-compose and addresses some runtime connection issues that arise from trying to start all the servers at once.

Ideally each container's app is able to survive the disappearance of other containers, perhaps those they depend on. One issue addressed here is the grafana http json server trying to make an RPC call prior to the hub registering the call with the crossbar server, but after the crossbar server is accepting connections. The result was a dead grafana http json server. We now handle the exception and wait a few seconds for the hub to finish starting up, then try again.

Similarly, the DataNodeServers could run into the same issue. I've solved this instead by having them wait a few seconds few seconds before starting, though a similar retry mechanism could be put in place in the DataNodeSever base class.

This works as is, and I think the connectivity issues addressed here are an improvement. I'm mostly opening this as a pull request to request others test it, following the updated README to see if it's clear and works well. I'm hoping this simplifies the use of Docker for those who aren't familiar with it, as labs are going to be expected to start up containers for local live monitoring.

ahincks commented 5 years ago

So, docker-compose build is giving me:

ERROR: Service 'sisock_crossbar' failed to build: pull access denied for sisock, repository does not exist or may require 'docker login'

Is it your intention that users be using docker login?

BrianJKoopman commented 5 years ago

Interesting. That's not my intention. Lemme remove all my images and try rebuilding to see if I can reproduce the error.

BrianJKoopman commented 5 years ago

Hmm, I can only reproduce the error if I make it so that the sisock image doesn't build (by commenting it out of the docker-compose.yaml file. Is 'sisock_crossbar' trying to build first on your system?

Docker should look for images on your local machine named 'sisock' first, then go out to the Docker Hub and look for things if no local image exists. So if the sisock image isn't build on your machine it will generate the error you're seeing. After you hit the error can you show me the output of docker image ls -a?

ahincks commented 5 years ago 
$ docker image ls -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
grafana/grafana     latest              920eb69ade2a        6 weeks ago         245MB
python              latest              a9d071760c82        6 weeks ago         923MB

Note that previous to running docker-compose build I had done:

$ docker network create --driver bridge sisock-net
eded8673f49e0cfa09985b250c25a48139801addded5c5fa8672de6c8ba8310d
$ docker run -d -p 3000:3000 --name=sisock_grafana -e "GF_INSTALL_PLUGINS=grafana-simple-json-datasource, natel-plotly-panel" grafana/grafana
ae0224aee08f2cec6aec8a55e5f0154603558d537489bdb53ea3af234ef2cd8f
$ docker network connect sisock-net sisock_grafana
BrianJKoopman commented 5 years ago

Yeah, the sisock image isn't built. Can you paste the full output of docker-compose build?

ahincks commented 5 years ago 
$ docker-compose build
Building sisock_crossbar
Step 1/5 : FROM sisock
ERROR: Service 'sisock_crossbar' failed to build: pull access denied for sisock, repository does not exist or may require 'docker login'
BrianJKoopman commented 5 years ago

Hmm, perhaps the order in the .yaml file isn't necessarily followed. I added a depends for sisock_crossbar on sisock. (It isn't clear in the docs this affects build order, but according to people on the internet it does...)

Try now?