Errors while building the Docker containers (calibration_server, object_position_server)

danieludick commented 5 years ago

I am currently trying to setup a calibration server for our TART telescope in SA. First off, I want to get the object_position_server going and tested on an Ubuntu machine. I am getting the following error after following the README.md file contained in that directory:

$ docker-compose build Docker machine "default" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.

For reference, I am unexperienced with Docker and might not have installed everything that is required. I did the following:

sudo snap install docker sudo apt install docker-compose

Is there anything I am missing?

danieludick commented 5 years ago

I seemed to have moved a bit further now by issuing:

sudo docker-compose build

But still I get that the build failed due to:

ERROR: Service 'object-position-server' failed to build: The command '/bin/sh -c apt-get install -y python-numpy python-dateutil' returned a non-zero code: 100

The full output are as follows:

Building object-position-server Step 1/16 : FROM debian:stretch-slim stretch-slim: Pulling from library/debian 6ae821421a7d: Pull complete Digest: sha256:772af16e8f2fa4a1dd64e022737a353c0ee58014ada0d330cf95ae0fc9271621 Status: Downloaded newer image for debian:stretch-slim ---> 9a4a82cec2d2 Step 2/16 : MAINTAINER Tim Molteno "tim@elec.ac.nz" ---> Running in a6dc0a7c82de Removing intermediate container a6dc0a7c82de ---> 917408173b5c Step 3/16 : ARG DEBIAN_FRONTEND=noninteractive ---> Running in 300a7b7df34d Removing intermediate container 300a7b7df34d ---> ab173a2c5d0d Step 4/16 : RUN apt-get update -y ---> Running in 86220a701259 Err:1 http://security.debian.org/debian-security stretch/updates InRelease Temporary failure resolving 'security.debian.org' Err:2 http://deb.debian.org/debian stretch InRelease Temporary failure resolving 'deb.debian.org' Err:3 http://deb.debian.org/debian stretch-updates InRelease Temporary failure resolving 'deb.debian.org' Reading package lists... W: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease Temporary failure resolving 'deb.debian.org' W: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/InRelease Temporary failure resolving 'security.debian.org' W: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/InRelease Temporary failure resolving 'deb.debian.org' W: Some index files failed to download. They have been ignored, or old ones used instead. Removing intermediate container 86220a701259 ---> 1372dfe67f50 Step 5/16 : RUN apt-get upgrade -y ---> Running in db9b9db4f7bd Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Removing intermediate container db9b9db4f7bd ---> 44f49bea14a7 Step 6/16 : RUN apt-get install -y python-numpy python-dateutil ---> Running in 2a17ae303aa1 Reading package lists... Building dependency tree... Reading state information... E: Unable to locate package python-numpy E: Unable to locate package python-dateutil ERROR: Service 'object-position-server' failed to build: The command '/bin/sh -c apt-get install -y python-numpy python-dateutil' returned a non-zero code: 100

tmolteno commented 5 years ago

The issues appear to be related to external web access on the machine. The docker image is unable to access the external repositories (deb.debian.org) to get the list of software. This means it can't find the numpy and dateutil packages.

More specifically, It seems that the DNS system on the computer running docker does not resolve the 'deb.debian.org' domain name. Could you check the output of the following command (assuming you're running some form of Linux)

wget http://deb.debian.org/debian/dists/stretch/

Your output should look something like...

$ wget http://deb.debian.org/debian/dists/stretch/
--2019-02-07 09:08:33--  http://deb.debian.org/debian/dists/stretch/
Resolving deb.debian.org (deb.debian.org)... 149.20.4.15, 128.31.0.62, 130.89.148.14, ...
Connecting to deb.debian.org (deb.debian.org)|149.20.4.15|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://cdn-fastly.deb.debian.org/debian/dists/stretch/ [following]
--2019-02-07 09:08:34--  http://cdn-fastly.deb.debian.org/debian/dists/stretch/
Resolving cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)... 151.101.232.204, 2a04:4e42:37::204
Connecting to cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)|151.101.232.204|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1801 (1.8K) [text/html]
Saving to: ‘index.html’

danieludick commented 5 years ago

Thanks @tmolteno . I ran now the command you suggested and got the following (seems to be what is expected):

--2019-02-07 07:24:17-- http://deb.debian.org/debian/dists/stretch/ Resolving deb.debian.org (deb.debian.org)... 130.89.148.14, 5.153.231.4, 128.31.0.62, ... Connecting to deb.debian.org (deb.debian.org)|130.89.148.14|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://cdn-fastly.deb.debian.org/debian/dists/stretch/ [following] --2019-02-07 07:24:18-- http://cdn-fastly.deb.debian.org/debian/dists/stretch/ Resolving cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)... 151.101.172.204, 2a04:4e42:29::204 Connecting to cdn-fastly.deb.debian.org (cdn-fastly.deb.debian.org)|151.101.172.204|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1801 (1.8K) [text/html] Saving to: ‘index.html’

index.html 100%[=====================================================================================================================================================================>] 1.76K --.-KB/s in 0s

2019-02-07 07:24:18 (175 MB/s) - ‘index.html’ saved [1801/1801]

I then tried the following again:

dludick@danie-ubuntu-desktop:/media/dludick/abf7b7f7-89e3-4ae4-99a5-77d7076d1892/work/stellenbosch/dev/TART/software/containers/object_position_server$ sudo docker-compose build [sudo] password for dludick: Building object-position-server Step 1/16 : FROM debian:stretch-slim ---> 9a4a82cec2d2 Step 2/16 : MAINTAINER Tim Molteno "tim@elec.ac.nz" ---> Using cache ---> 917408173b5c Step 3/16 : ARG DEBIAN_FRONTEND=noninteractive ---> Using cache ---> ab173a2c5d0d Step 4/16 : RUN apt-get update -y ---> Using cache ---> 1372dfe67f50 Step 5/16 : RUN apt-get upgrade -y ---> Using cache ---> 44f49bea14a7 Step 6/16 : RUN apt-get install -y python-numpy python-dateutil ---> Running in 2818ad556fae Reading package lists... Building dependency tree... Reading state information... E: Unable to locate package python-numpy E: Unable to locate package python-dateutil ERROR: Service 'object-position-server' failed to build: The command '/bin/sh -c apt-get install -y python-numpy python-dateutil' returned a non-zero code: 100

danieludick commented 5 years ago

This seems to be a networking/firewall issue with our University. I tried now with another Ubuntu PC connected to the internet and was able to get the following output after issuing:

docker-compose build

Output:

building object-position-server Step 1/16 : FROM debian:stretch-slim stretch-slim: Pulling from library/debian 6ae821421a7d: Pull complete Digest: sha256:772af16e8f2fa4a1dd64e022737a353c0ee58014ada0d330cf95ae0fc9271621 Status: Downloaded newer image for debian:stretch-slim ---> 9a4a82cec2d2 Step 2/16 : MAINTAINER Tim Molteno "tim@elec.ac.nz" ---> Running in 2dd9273b7702 Removing intermediate container 2dd9273b7702 ---> dd850878c344 Step 3/16 : ARG DEBIAN_FRONTEND=noninteractive ---> Running in 9bf8a655654e Removing intermediate container 9bf8a655654e ---> 2620981350d5 Step 4/16 : RUN apt-get update -y ---> Running in 86c6b1acd179 Get:1 http://security-cdn.debian.org/debian-security stretch/updates InRelease [94.3 kB] Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease Get:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB] Get:5 http://security-cdn.debian.org/debian-security stretch/updates/main amd64 Packages [475 kB] Get:4 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB] Get:6 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages [7748 B] Get:7 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B] Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 Packages [7090 kB] Fetched 7878 kB in 10s (749 kB/s) Reading package lists... Removing intermediate container 86c6b1acd179 ---> 4499a1b5ea73 Step 5/16 : RUN apt-get upgrade -y ---> Running in 558853c10998 Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Removing intermediate container 558853c10998 ... ---> 54dca4ab3cbc Step 16/16 : CMD ["python", "restful_api.py", "--host=0.0.0.0"] ---> Running in af7f4f9f852b Removing intermediate container af7f4f9f852b ---> e5f18a459c15 Successfully built e5f18a459c15 Successfully tagged objectpositionserver_object-position-server:latest

Then following the steps in the README.md I did the following:

dludick@DLUDICK-W540:/media/dludick/work/stellenbosch/research/development/radio_astronomy/TART/software/containers/object_position_server$ docker-compose up -d Creating network "objectpositionserver_default" with the default driver Creating objectpositionserver_object-position-server_1 dludick@DLUDICK-W540:/media/dludick/work/stellenbosch/research/development/radio_astronomy/TART/software/containers/object_position_server$ docker attach ops Error: No such container: ops

Not sure if this error is really an issue? I tested it as follows:

wget -qO- "http://localhost:8876/catalog?lat=-33.94&lon=18.86"[ { "az": 306.685402, "el": 35.539246, "jy": 1500000.0, "name": "AOR-E (EGNOS/PRN 120)", "r": 38192337.5 }, .. }, { "az": 47.347709, "el": 64.696603, "jy": 10000.0, "name": "sun", "r": 10000000000.0 } ]

So it seems to be working. Not sure the networking issue is worth trying to resolve. I am going to continue with the calibration server install now on this PC outside of the network that gives issues, and then try and connect it to the Univ. network afterwards to calibrate the TART system we have here.

danieludick commented 5 years ago

Started now with the calibration_server (also on the same host where the object detection server is running). Running in to the following error when running sh build.sh:

Package python-healpy is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source

E: Package 'python-astropy' has no installation candidate E: Package 'python-healpy' has no installation candidate The command '/bin/sh -c apt-get install -y python-pip python-numpy python-dateutil python-matplotlib python-scipy python-astropy python-healpy' returned a non-zero code: 100

tmolteno commented 5 years ago

Thanks for the report. It looks like the docker image was based on a testing distribution of debian where those packages had been temporarily disabled. I'm curently testing a fix to base it off a stable version.

tmolteno commented 5 years ago

@danieludick Could you please pull the latest version, it seems to fix the issue here at least.

danieludick commented 5 years ago

Thanks @tmolteno . Pulling the latest version seemd to work :). The calibration is still running. I will close the issue once I can confirm that everything works as expected.

tmolteno commented 5 years ago

@danieludick I have made a few documentation changes to try and avoid duplicating instructions for software installation, and also to make the flow a little more logical.

danieludick commented 5 years ago

Just a note on this issue:

To get the Docker installation sorted out on Ubuntu 18.02, one should follow steps as outlined on https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04, as the Docker installation package available in the official Ubuntu repository may not be the latest version.
Also for the Object position server, I had to change the following in the Dockerfile:

FROM debian:stretch-slim

FROM debian:stretch

Then everything built. The servers are now installed on my machine and I am going to close this issue.

tmolteno / TART

Errors while building the Docker containers (calibration_server, object_position_server) #36

FROM debian:stretch-slim