search

CrunchyData / crunchy-postgresql-manager

PostgreSQL-as-a-Service Management Utility. Superseded by Crunchy Container Suite: https://github.com/CrunchyData/crunchy-containers
Apache License 2.0
36 stars 2 forks source link

Does this work on a Mac? #121

Closed MadBomber closed 8 years ago

MadBomber commented 8 years ago

I cloned this repo to my fully-loaded developer Mac. I am following the instructions in chapter 3 "developer setup" which non-nonsensically says:

mkdir cpmproject
cp Vagrantfile .
vagrant up

So what I did was make my cpmproject directory outside of the clone's workspace and did this:

mkdir cpmproject
cd cpmproject
cp ../crunchy-postgresql-manager/Vagrantfile .
vagrant up

In the process of building the VirtualBox VM the log show 404 errors on rpms making me think that the Vagrantfile might be out of date. Then there is the problems at the end trying to find host names that do not exist.

If this should have worked on a Mac, I will include the entire log on the follow up....

Dewayne o-*

jmccormick2001 commented 8 years ago

hmmn, not sure, I've never built it or run it on a Mac. Only centos/rhel. very interested in your steps to get it built/running on mac, would be happy to know how you do it.

MadBomber commented 8 years ago

From my point of view the host that runs the VM is moot. The VM contains all that is necessary to run the application.

The log file is long ~19,000 lines. The only issues during the build are these few; however, they seem to be critical to the operation.

<snip>
==> default: Step 3 : RUN rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
==> default:  ---> Running in 1b2769ea5103
==> default: curl: (22) The requested URL returned error: 404 Not Found
==> default:
==> default: Retrieving http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
==> default: error: skipping http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm - transfer failed
==> default:
==> default: The command '/bin/sh -c rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm' returned a non-zero code: 1
==> default: make[1]: Leaving directory `/home/vagrant/devproject/src/github.com/crunchydata/crunchy-postgresql-manager/images/cpm-server'
==> default: make[1]: *** [build] Error 1
==> default: make: *** [buildimages] Error 2
==> default: set env vars used by installation scripts
==> default: configure docker

<snip -- here is the last part of the log; I am assuming that these containers are being build in the guest OS not the host>

==> default: Error response from daemon: Container skybridge not found
==> default: Error: failed to remove containers: [skybridge]
==> default: 2c4e2ac0b390645ae3fa6109525ce275a8d1b2c552e5800dd2892c26cea90941
==> default: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
==> default: run cpm-server container
==> default: restarting cpm-server
==> default: Error response from daemon: no such id: cpm-server
==> default: Error: failed to stop containers: [cpm-server]
==> default: Error response from daemon: no such id: cpm-server
==> default: Error: failed to remove containers: [cpm-server]
==> default: time="2016-05-23T12:29:59-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:29:59-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:30:19-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:30:19-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:30:35-04:00" level=error msg="HTTP error: Error: image crunchydata/cpm-server:latest not found" status=500
==> default: run cpm app containers
==> default: Error response from daemon: Error: image crunchydata/cpm-server:latest not found
==> default: setting up keys dir...
==> default: mkdir: cannot create directory ‘/var/cpm/keys’: File exists
==> default: setting up keys dir...
==> default: mkdir: cannot create directory ‘/var/cpm/keys’: File exists
==> default: setting up log dir...
==> default: mkdir: cannot create directory ‘/var/cpm/logs’: File exists
==> default: restarting cpm-web container...
==> default: time="2016-05-23T12:30:35-04:00" level=error msg="HTTP error: No such container: cpm-web" status=404
==> default: Error response from daemon: No such container: cpm-web
==> default: Error: failed to stop containers: [cpm-web]
==> default: time="2016-05-23T12:30:35-04:00" level=error msg="HTTP error: Container cpm-web not found" status=404
==> default: Error response from daemon: Container cpm-web not found
==> default: Error: failed to remove containers: [cpm-web]
==> default: time="2016-05-23T12:30:39-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:30:39-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:30:59-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:30:59-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:31:14-04:00" level=error msg="HTTP error: Error: image crunchydata/cpm:latest not found" status=500
==> default: Error response from daemon: Error: image crunchydata/cpm:latest not found
==> default: restarting cpm-admin container...
==> default: time="2016-05-23T12:31:17-04:00" level=error msg="HTTP error: No such container: cpm-admin" status=404
==> default: Error response from daemon: No such container: cpm-admin
==> default: Error: failed to stop containers: [cpm-admin]
==> default: time="2016-05-23T12:31:17-04:00" level=error msg="HTTP error: Container cpm-admin not found" status=404
==> default: Error response from daemon: Container cpm-admin not found
==> default: Error: failed to remove containers: [cpm-admin]
==> default: time="2016-05-23T12:31:19-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:31:19-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:31:39-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:31:39-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:31:56-04:00" level=error msg="HTTP error: Error: image crunchydata/cpm-admin:latest not found" status=500
==> default: Error response from daemon: Error: image crunchydata/cpm-admin:latest not found
==> default: restarting cpm-task container...
==> default: time="2016-05-23T12:31:58-04:00" level=error msg="HTTP error: No such container: cpm-task" status=404
==> default: Error response from daemon: No such container: cpm-task
==> default: Error: failed to stop containers: [cpm-task]
==> default: time="2016-05-23T12:31:58-04:00" level=error msg="HTTP error: Container cpm-task not found" status=404
==> default: Error response from daemon: Container cpm-task not found
==> default: Error: failed to remove containers: [cpm-task]
==> default: time="2016-05-23T12:31:59-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:31:59-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:32:19-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:32:19-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:32:38-04:00" level=error msg="HTTP error: Error: image crunchydata/cpm-task:latest not found" status=500
==> default: Error response from daemon: Error: image crunchydata/cpm-task:latest not found
==> default: restarting cpm-promdash container...
==> default: time="2016-05-23T12:32:39-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:32:39-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:32:40-04:00" level=error msg="HTTP error: No such container: cpm-promdash" status=404
==> default: Error response from daemon: No such container: cpm-promdash
==> default: Error: failed to stop containers: [cpm-promdash]
==> default: time="2016-05-23T12:32:40-04:00" level=error msg="HTTP error: Container cpm-promdash not found" status=404
==> default: Error response from daemon: Container cpm-promdash not found
==> default: Error: failed to remove containers: [cpm-promdash]
==> default: af1b577903e7de50d23947bac548b568ebf190036fa576360840e715bc0a2927
==> default: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
==> default: restarting cpm-prometheus container...
==> default: chmod: cannot access ‘/var/cpm/config/prometheus.yml’: No such file or directory
==> default: chcon: cannot access ‘/var/cpm/config/prometheus.yml’: No such file or directory
==> default: time="2016-05-23T12:32:43-04:00" level=error msg="HTTP error: No such container: cpm-prometheus" status=404
==> default: Error response from daemon: No such container: cpm-prometheus
==> default: Error: failed to stop containers: [cpm-prometheus]
==> default: time="2016-05-23T12:32:43-04:00" level=error msg="HTTP error: Container cpm-prometheus not found" status=404
==> default: Error response from daemon: Container cpm-prometheus not found
==> default: Error: failed to remove containers: [cpm-prometheus]
==> default: 2b0932c56912bd7df86a7740f9bbd86279245feee662a543810441cc1089d511
==> default: Error response from daemon: Cannot start container 2b0932c56912bd7df86a7740f9bbd86279245feee662a543810441cc1089d511: [8] System error: not a directory
==> default: sleeping a bit while cpm-prometheus starts up....
==> default: restarting cpm-collect container...
==> default: time="2016-05-23T12:32:54-04:00" level=error msg="HTTP error: No such container: cpm-collect" status=404
==> default: Error response from daemon: No such container: cpm-collect
==> default: Error: failed to stop containers: [cpm-collect]
==> default: time="2016-05-23T12:32:54-04:00" level=error msg="HTTP error: Container cpm-collect not found" status=404
==> default: Error response from daemon: Container cpm-collect not found
==> default: Error: failed to remove containers: [cpm-collect]
==> default: time="2016-05-23T12:32:59-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:32:59-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:33:19-04:00" level=info msg="Registering on the discovery service every 20s..." addr="10.0.2.15:2375" discovery="file:///var/cpm/data/swarm_cluster_file"
==> default: time="2016-05-23T12:33:19-04:00" level=error msg="not implemented in this discovery service"
==> default: time="2016-05-23T12:33:34-04:00" level=error msg="HTTP error: Error: image crunchydata/cpm-collect:latest not found" status=500
==> default: testing containers for DNS resolution....
==> default: Error response from daemon: Error: image crunchydata/cpm-collect:latest not found
==> default: ping: unknown host cpm-web.crunchy.lab
==> default: ping: unknown host cpm-admin.crunchy.lab
==> default: ping: unknown host cpm-task.crunchy.lab
==> default: PING cpm-promdash.crunchy.lab (172.17.0.2) 56(84) bytes of data.
==> default: 64 bytes from cpm-promdash.crunchy.lab (172.17.0.2): icmp_seq=1 ttl=64 time=0.060 ms
==> default: 64 bytes from cpm-promdash.crunchy.lab (172.17.0.2): icmp_seq=2 ttl=64 time=0.047 ms
==> default:
==> default: --- cpm-promdash.crunchy.lab ping statistics ---
==> default: 2 packets transmitted, 2 received, 0% packet loss, time 1001ms
==> default: rtt min/avg/max/mdev = 0.047/0.053/0.060/0.009 ms
==> default: ping: unknown host cpm-prometheus.crunchy.lab
==> default: ping: unknown host cpm-collect.crunchy.lab
==> default: ping: unknown host cpm-server.crunchy.lab
jmccormick2001 commented 8 years ago

ah, the Dockerfiles are using the old (and now missing) epel release RPM path it appears.

Let me take a look at fixing this.

jmccormick2001 commented 8 years ago

I've updated those scripts which should hopefully move you along further, give it a try. Changes on are master.

MadBomber commented 8 years ago

I think it might have worked... silly question: How do I know it worked? I am assuming that I can got to a URL from my Mac (host) to access the CPM app running in a guest VM but I'm too lazy////tired to look it up in the documentation.

The vagrant up command has been idle for some time. The last two lines on the console are:

==> default: Step 11 : RUN fluent-gem install fluent-plugin-elasticsearch --no-ri --no-rdoc ==> default: ---> Running in 1f40f7454a24

On Mon, May 23, 2016 at 4:33 PM, Jeff McCormick notifications@github.com wrote:

I've updated those scripts which should hopefully move you along further, give it a try. Changes on are master.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221103001

Dewayne o-*

jmccormick2001 commented 8 years ago

see if you can 'vagrant ssh' into the VM, if so, you are on your way, from there you can start running various CPM containers.

MadBomber commented 8 years ago

yes I can ssh into the vagrant image. Was only the OS built? Where should I pick up in the dev-setup docs? I did verify within the guest OS that the OPTIONS variable in the /etc/sysconfig/docker file matches the dev-setup says it should be.

MadBomber commented 8 years ago

On a whim I did this within the guest OS:

[vagrant@cpm crunchy-postgresql-manager]$ sudo ./run-cpm.sh
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
cp: cannot stat ‘/sbin/key.pem’: No such file or directory
cp: cannot stat ‘/sbin/cert.pem’: No such file or directory
./run-cpm.sh: line 55: /images/cpm/run-cpm-web.sh: No such file or directory
./run-cpm.sh: line 57: /images/cpm-admin/run-cpm-admin.sh: No such file or directory
./run-cpm.sh: line 59: /images/cpm-task/run-cpm-task.sh: No such file or directory
./run-cpm.sh: line 61: /images/cpm-prometheus/run-cpm-prometheus.sh: No such file or directory
sleeping a bit while cpm-prometheus starts up....
./run-cpm.sh: line 66: /images/cpm-collect/run-cpm-collect.sh: No such file or directory
testing containers for DNS resolution....
ping: unknown host cpm-web.
ping: unknown host cpm-admin.
ping: unknown host cpm-task.
ping: unknown host cpm-promdash.
ping: unknown host cpm-prometheus.
ping: unknown host cpm-collect.
ping: unknown host cpm-server.

Is this confirmation that all I have is an OS and I need to pickup the dev-setup process at make cpm?

MadBomber commented 8 years ago

hold on a minute. Things look like they were built. I'm thinking that it might be a problem with not having some system environment variables set. I'm adding the following to the vagrant user's .bashrc file:

export GOPATH=/home/vagrant/devproject
export GOBIN=$GOPATH/bin
export PATH=$GOBIN:$PATH
export CPMROOT=$GOPATH/src/github.com/crunchydata/crunchy-postgresql-manager

Still looking at the new toy. I will document whatever changes I have to make in this issue.

jmccormick2001 commented 8 years ago

the Vagrantfile runs this script:

https://github.com/CrunchyData/crunchy-postgresql-manager/blob/master/sbin/vagrant-setup.sh

that script does all the dependency setup on the box, configures docker/swarm, builds the images, sets up directories, etc.

jmccormick2001 commented 8 years ago

your on the right path...I have another emerging project, crunchy-containers, that takes most of this functionality and runs it under kubernetes and Openshift, let me know if you have any interest in that and I'll add you to the list of collaborators.

On Wed, May 25, 2016 at 11:05 AM, Dewayne VanHoozer < notifications@github.com> wrote:

hold on a minute. Things look like they were built. I'm thinking that it might be a problem with not having some system environment variables set. I'm adding the following to the cpm user's .bashrc file:

export GOPATH=~/devprojectexport GOBIN=$GOPATH/binexport PATH=$GOBIN:$PATHexport CPMROOT=$GOPATH/src/github.com/crunchydata/crunchy-postgresql-manager

Still looking at the new toy. I will document whatever changes I have to make in this issue.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221621139

MadBomber commented 8 years ago

Don't know how much help I can be on the crunchy-containers project, but it is something in which I am interested. Sure sign me up.

w/r/t current issue, looking around things look like there were built. There were some OS depencies that were not applied during the initial vagrant up process. I've tried executing some of the run scripts as user cpm and get told that I must do it as root. So I sudo the run script and get told that some system environment variables are not set. Working through that now. So far is just CPMROOT and one for docker which I failed to document.... gotta find it again.

MadBomber commented 8 years ago

I'm hitting the wall with the error "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

I started the docker.service.

[vagrant@cpm crunchy-postgresql-manager]$ sudo systemctl enable docker.service
[vagrant@cpm crunchy-postgresql-manager]$ sudo systemctl start docker.service
[vagrant@cpm crunchy-postgresql-manager]$ sudo systemctl | fgrep -i docker
  sys-devices-virtual-net-docker0.device                                                    loaded active plugged   /sys/devices/virtual/net/docker0
  sys-subsystem-net-devices-docker0.device                                                  loaded active plugged   /sys/subsystem/net/devices/docker0
  var-lib-docker-devicemapper.mount                                                         loaded active mounted   /var/lib/docker/devicemapper
● docker-storage-setup.service                                                              loaded failed failed    Docker Storage Setup

Is there something else that needs to be done to get the docker daemon running?

jmccormick2001 commented 8 years ago

as root do this:

groupadd docker usermod -a -G docker vagrant systemctl restart docker.service

then as the vagrant user, you should be able to do docker commands directly.

On Wed, May 25, 2016 at 12:34 PM, Dewayne VanHoozer < notifications@github.com> wrote:

I'm hitting the wall with the error "Cannot connect to the Docker daemon. Is the docker daemon running on this host?"

I started the docker.service.

[vagrant@cpm crunchy-postgresql-manager]$ sudo systemctl enable docker.service [vagrant@cpm crunchy-postgresql-manager]$ sudo systemctl start docker.service [vagrant@cpm crunchy-postgresql-manager]$ sudo systemctl | fgrep -i docker sys-devices-virtual-net-docker0.device loaded active plugged /sys/devices/virtual/net/docker0 sys-subsystem-net-devices-docker0.device loaded active plugged /sys/subsystem/net/devices/docker0 var-lib-docker-devicemapper.mount loaded active mounted /var/lib/docker/devicemapper ● docker-storage-setup.service loaded failed failed Docker Storage Setup

Is there something else that needs to be done to get the docker daemon running?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221646823

MadBomber commented 8 years ago

Its starting to look like the docker containers were not built during the initial vagrant up process. Got that going now as root. I will make the group assignment and restart the service as you suggested after the current build has completed.

MadBomber commented 8 years ago

I also have to run 'docker daemon &' ... images are built and 'docker images' gives an interesting list:

[root@cpm crunchy-postgresql-manager]# docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
cpm-efk                                latest              9deb7d19dada        6 minutes ago       617.4 MB
crunchydata/cpm-efk                    latest              9deb7d19dada        6 minutes ago       617.4 MB
crunchydata/cpm-collect                latest              56757506b3a3        37 minutes ago      252.3 MB
cpm-collect                            latest              56757506b3a3        37 minutes ago      252.3 MB
cpm-backrest-restore-job               latest              dc174ce3d737        37 minutes ago      250.9 MB
crunchydata/cpm-backrest-restore-job   latest              dc174ce3d737        37 minutes ago      250.9 MB
cpm-restore-job                        latest              5103c5b3a714        38 minutes ago      262.3 MB
crunchydata/cpm-restore-job            latest              5103c5b3a714        38 minutes ago      262.3 MB
crunchydata/cpm-backup-job             latest              7ddb3619937c        38 minutes ago      261.9 MB
cpm-backup-job                         latest              7ddb3619937c        38 minutes ago      261.9 MB
cpm-task                               latest              024d7f1c5417        38 minutes ago      252 MB
crunchydata/cpm-task                   latest              024d7f1c5417        38 minutes ago      252 MB
cpm-pgpool                             latest              afec750da4e3        38 minutes ago      253.8 MB
crunchydata/cpm-pgpool                 latest              afec750da4e3        38 minutes ago      253.8 MB
crunchydata/cpm-node-proxy             latest              7d18dce04496        39 minutes ago      301.2 MB
cpm-node-proxy                         latest              7d18dce04496        39 minutes ago      301.2 MB
crunchydata/cpm-node                   latest              af01bf8a6aa3        39 minutes ago      310.5 MB
cpm-node                               latest              af01bf8a6aa3        39 minutes ago      310.5 MB
crunchydata/cpm                        latest              61d5372e63c7        40 minutes ago      234 MB
cpm                                    latest              61d5372e63c7        40 minutes ago      234 MB
cpm-admin                              latest              7583e184177c        40 minutes ago      305.8 MB
crunchydata/cpm-admin                  latest              7583e184177c        40 minutes ago      305.8 MB
cpm-server                             latest              b6f6d91c0244        40 minutes ago      328.8 MB
crunchydata/cpm-server                 latest              b6f6d91c0244        40 minutes ago      328.8 MB
<none>                                 <none>              9ed80b4aaf49        23 hours ago        601.9 MB
<none>                                 <none>              611e8e1f1ed5        23 hours ago        252.3 MB
<none>                                 <none>              9fc30cccc2d7        23 hours ago        250.9 MB
<none>                                 <none>              43a48a80d799        24 hours ago        262.3 MB
<none>                                 <none>              ed804d31159b        24 hours ago        261.9 MB
<none>                                 <none>              48a63c5c069b        24 hours ago        252 MB
<none>                                 <none>              0412c379fb0b        24 hours ago        253.8 MB
<none>                                 <none>              bd98562700fb        24 hours ago        301.2 MB
<none>                                 <none>              7628323efc51        24 hours ago        310.5 MB
<none>                                 <none>              27e44e73516c        24 hours ago        234 MB
<none>                                 <none>              8afa151776c8        24 hours ago        305.8 MB
<none>                                 <none>              6eef6e5cb3e3        24 hours ago        328.8 MB
docker.io/centos                       7                   8c59c0a396b7        8 days ago          196.7 MB
docker.io/prom/prometheus              latest              4cbd7a0f15be        2 weeks ago         58.48 MB
docker.io/prom/promdash                latest              e05cec4b67dc        3 weeks ago         180.8 MB
[root@cpm crunchy-postgresql-manager]#

My guess is that all the images label "" were created during the initial vagrant up process. Their sizes are consisten with the cpm component images.

MadBomber commented 8 years ago

This should have worked. Looks like a combination of docker and DNS problems.

[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]# source cpmenv
[root@cpm crunchy-postgresql-manager]# cat cpmenv

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
#export CPMROOT=$DIR/src/github.com/crunchydata/crunchy-postgresql-manager
export CPMROOT=$DIR
export LOCAL_IP=`/sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | tail -1`
export SWARM_MANAGER_URL=tcp://$LOCAL_IP:8000
export CPM_DOMAIN=crunchy.lab
export FLUENT_URL=$LOCAL_IP:24224
export KEYSDIR=/var/cpm/keys
export CPM_DOMAIN=crunchy.lab
[root@cpm crunchy-postgresql-manager]# echo $CPMROOT
/home/vagrant/devproject/src/github.com/crunchydata/crunchy-postgresql-manager
[root@cpm crunchy-postgresql-manager]# echo $LOCAL_IP
10.0.2.15
[root@cpm crunchy-postgresql-manager]# echo $SWARM_MANAGER_URL
tcp://10.0.2.15:8000
[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]# ./run-cpm.sh
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
setting up log dir...
mkdir: cannot create directory ‘/var/cpm/logs’: File exists
restarting cpm-web container...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to stop containers: [cpm-web]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to remove containers: [cpm-web]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
restarting cpm-admin container...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to stop containers: [cpm-admin]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to remove containers: [cpm-admin]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
restarting cpm-task container...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to stop containers: [cpm-task]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to remove containers: [cpm-task]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
restarting cpm-promdash container...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to stop containers: [cpm-promdash]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to remove containers: [cpm-promdash]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
restarting cpm-prometheus container...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to stop containers: [cpm-prometheus]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to remove containers: [cpm-prometheus]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
sleeping a bit while cpm-prometheus starts up....
restarting cpm-collect container...
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to stop containers: [cpm-collect]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Error: failed to remove containers: [cpm-collect]
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
testing containers for DNS resolution....
ping: unknown host cpm-web.crunchy.lab
ping: unknown host cpm-admin.crunchy.lab
ping: unknown host cpm-task.crunchy.lab
ping: unknown host cpm-promdash.crunchy.lab
ping: unknown host cpm-prometheus.crunchy.lab
ping: unknown host cpm-collect.crunchy.lab
ping: unknown host cpm-server.crunchy.lab
[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]# docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
cpm-efk                                latest              9deb7d19dada        About an hour ago   617.4 MB
crunchydata/cpm-efk                    latest              9deb7d19dada        About an hour ago   617.4 MB
crunchydata/cpm-collect                latest              56757506b3a3        2 hours ago         252.3 MB
cpm-collect                            latest              56757506b3a3        2 hours ago         252.3 MB
cpm-backrest-restore-job               latest              dc174ce3d737        2 hours ago         250.9 MB
crunchydata/cpm-backrest-restore-job   latest              dc174ce3d737        2 hours ago         250.9 MB
cpm-restore-job                        latest              5103c5b3a714        2 hours ago         262.3 MB
crunchydata/cpm-restore-job            latest              5103c5b3a714        2 hours ago         262.3 MB
cpm-backup-job                         latest              7ddb3619937c        2 hours ago         261.9 MB
crunchydata/cpm-backup-job             latest              7ddb3619937c        2 hours ago         261.9 MB
cpm-task                               latest              024d7f1c5417        2 hours ago         252 MB
crunchydata/cpm-task                   latest              024d7f1c5417        2 hours ago         252 MB
cpm-pgpool                             latest              afec750da4e3        2 hours ago         253.8 MB
crunchydata/cpm-pgpool                 latest              afec750da4e3        2 hours ago         253.8 MB
cpm-node-proxy                         latest              7d18dce04496        2 hours ago         301.2 MB
crunchydata/cpm-node-proxy             latest              7d18dce04496        2 hours ago         301.2 MB
cpm-node                               latest              af01bf8a6aa3        2 hours ago         310.5 MB
crunchydata/cpm-node                   latest              af01bf8a6aa3        2 hours ago         310.5 MB
cpm                                    latest              61d5372e63c7        2 hours ago         234 MB
crunchydata/cpm                        latest              61d5372e63c7        2 hours ago         234 MB
cpm-admin                              latest              7583e184177c        2 hours ago         305.8 MB
crunchydata/cpm-admin                  latest              7583e184177c        2 hours ago         305.8 MB
cpm-server                             latest              b6f6d91c0244        2 hours ago         328.8 MB
crunchydata/cpm-server                 latest              b6f6d91c0244        2 hours ago         328.8 MB
<none>                                 <none>              9ed80b4aaf49        25 hours ago        601.9 MB
<none>                                 <none>              611e8e1f1ed5        25 hours ago        252.3 MB
<none>                                 <none>              9fc30cccc2d7        25 hours ago        250.9 MB
<none>                                 <none>              43a48a80d799        25 hours ago        262.3 MB
<none>                                 <none>              ed804d31159b        25 hours ago        261.9 MB
<none>                                 <none>              48a63c5c069b        25 hours ago        252 MB
<none>                                 <none>              0412c379fb0b        25 hours ago        253.8 MB
<none>                                 <none>              bd98562700fb        25 hours ago        301.2 MB
<none>                                 <none>              7628323efc51        25 hours ago        310.5 MB
<none>                                 <none>              27e44e73516c        25 hours ago        234 MB
<none>                                 <none>              8afa151776c8        25 hours ago        305.8 MB
<none>                                 <none>              6eef6e5cb3e3        25 hours ago        328.8 MB
docker.io/centos                       7                   8c59c0a396b7        8 days ago          196.7 MB
docker.io/prom/prometheus              latest              4cbd7a0f15be        2 weeks ago         58.48 MB
docker.io/prom/promdash                latest              e05cec4b67dc        3 weeks ago         180.8 MB
[root@cpm crunchy-postgresql-manager]#
[root@cpm crunchy-postgresql-manager]# echo wtf over
wtf over
[root@cpm crunchy-postgresql-manager]#
jmccormick2001 commented 8 years ago

it appears swarm is not running for some reason

On Wed, May 25, 2016 at 3:09 PM, Dewayne VanHoozer <notifications@github.com

wrote:

This should have worked. Looks like a combination of docker and DNS problems.

[root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# source cpmenv [root@cpm crunchy-postgresql-manager]# cat cpmenv

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"#export CPMROOT=$DIR/src/github.com/crunchydata/crunchy-postgresql-managerexport CPMROOT=$DIRexport LOCAL_IP=/sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | tail -1export SWARM_MANAGER_URL=tcp://$LOCAL_IP:8000export CPM_DOMAIN=crunchy.labexport FLUENT_URL=$LOCAL_IP:24224export KEYSDIR=/var/cpm/keysexport CPM_DOMAIN=crunchy.lab [root@cpm crunchy-postgresql-manager]# echo $CPMROOT /home/vagrant/devproject/src/github.com/crunchydata/crunchy-postgresql-manager [root@cpm crunchy-postgresql-manager]# echo $LOCAL_IP 10.0.2.15 [root@cpm crunchy-postgresql-manager]# echo $SWARM_MANAGER_URL tcp://10.0.2.15:8000 [root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# ./run-cpm.sh setting up keys dir... mkdir: cannot create directory ‘/var/cpm/keys’: File exists setting up keys dir... mkdir: cannot create directory ‘/var/cpm/keys’: File exists setting up log dir... mkdir: cannot create directory ‘/var/cpm/logs’: File exists restarting cpm-web container... Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to stop containers: [cpm-web] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to remove containers: [cpm-web] Cannot connect to the Docker daemon. Is the docker daemon running on this host? restarting cpm-admin container... Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to stop containers: [cpm-admin] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to remove containers: [cpm-admin] Cannot connect to the Docker daemon. Is the docker daemon running on this host? restarting cpm-task container... Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to stop containers: [cpm-task] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to remove containers: [cpm-task] Cannot connect to the Docker daemon. Is the docker daemon running on this host? restarting cpm-promdash container... Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to stop containers: [cpm-promdash] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to remove containers: [cpm-promdash] Cannot connect to the Docker daemon. Is the docker daemon running on this host? restarting cpm-prometheus container... Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to stop containers: [cpm-prometheus] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to remove containers: [cpm-prometheus] Cannot connect to the Docker daemon. Is the docker daemon running on this host? sleeping a bit while cpm-prometheus starts up.... restarting cpm-collect container... Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to stop containers: [cpm-collect] Cannot connect to the Docker daemon. Is the docker daemon running on this host? Error: failed to remove containers: [cpm-collect] Cannot connect to the Docker daemon. Is the docker daemon running on this host? testing containers for DNS resolution.... ping: unknown host cpm-web.crunchy.lab ping: unknown host cpm-admin.crunchy.lab ping: unknown host cpm-task.crunchy.lab ping: unknown host cpm-promdash.crunchy.lab ping: unknown host cpm-prometheus.crunchy.lab ping: unknown host cpm-collect.crunchy.lab ping: unknown host cpm-server.crunchy.lab [root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE cpm-efk latest 9deb7d19dada About an hour ago 617.4 MB crunchydata/cpm-efk latest 9deb7d19dada About an hour ago 617.4 MB crunchydata/cpm-collect latest 56757506b3a3 2 hours ago 252.3 MB cpm-collect latest 56757506b3a3 2 hours ago 252.3 MB cpm-backrest-restore-job latest dc174ce3d737 2 hours ago 250.9 MB crunchydata/cpm-backrest-restore-job latest dc174ce3d737 2 hours ago 250.9 MB cpm-restore-job latest 5103c5b3a714 2 hours ago 262.3 MB crunchydata/cpm-restore-job latest 5103c5b3a714 2 hours ago 262.3 MB cpm-backup-job latest 7ddb3619937c 2 hours ago 261.9 MB crunchydata/cpm-backup-job latest 7ddb3619937c 2 hours ago 261.9 MB cpm-task latest 024d7f1c5417 2 hours ago 252 MB crunchydata/cpm-task latest 024d7f1c5417 2 hours ago 252 MB cpm-pgpool latest afec750da4e3 2 hours ago 253.8 MB crunchydata/cpm-pgpool latest afec750da4e3 2 hours ago 253.8 MB cpm-node-proxy latest 7d18dce04496 2 hours ago 301.2 MB crunchydata/cpm-node-proxy latest 7d18dce04496 2 hours ago 301.2 MB cpm-node latest af01bf8a6aa3 2 hours ago 310.5 MB crunchydata/cpm-node latest af01bf8a6aa3 2 hours ago 310.5 MB cpm latest 61d5372e63c7 2 hours ago 234 MB crunchydata/cpm latest 61d5372e63c7 2 hours ago 234 MB cpm-admin latest 7583e184177c 2 hours ago 305.8 MB crunchydata/cpm-admin latest 7583e184177c 2 hours ago 305.8 MB cpm-server latest b6f6d91c0244 2 hours ago 328.8 MB crunchydata/cpm-server latest b6f6d91c0244 2 hours ago 328.8 MB 9ed80b4aaf49 25 hours ago 601.9 MB 611e8e1f1ed5 25 hours ago 252.3 MB 9fc30cccc2d7 25 hours ago 250.9 MB 43a48a80d799 25 hours ago 262.3 MB ed804d31159b 25 hours ago 261.9 MB 48a63c5c069b 25 hours ago 252 MB 0412c379fb0b 25 hours ago 253.8 MB bd98562700fb 25 hours ago 301.2 MB 7628323efc51 25 hours ago 310.5 MB 27e44e73516c 25 hours ago 234 MB 8afa151776c8 25 hours ago 305.8 MB 6eef6e5cb3e3 25 hours ago 328.8 MBdocker.io/centos 7 8c59c0a396b7 8 days ago 196.7 MBdocker.io/prom/prometheus latest 4cbd7a0f15be 2 weeks ago 58.48 MBdocker.io/prom/promdash latest e05cec4b67dc 3 weeks ago 180.8 MB [root@cpm crunchy-postgresql-manager]# [root@cpm crunchy-postgresql-manager]# echo wtf over wtf over [root@cpm crunchy-postgresql-manager]#

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221692059

MadBomber commented 8 years ago

I started docker daemon first then swarm. Got a connection error on port 2375. Containers still not resolved.

# running as root in $CPMROOT after source cpmenv

[root@cpm crunchy-postgresql-manager]# alias psf
alias psf='ps aux | f $@'
[root@cpm crunchy-postgresql-manager]# alias f
alias f='fgrep -i'

[root@cpm crunchy-postgresql-manager]# psf swarm
root      5595  0.0  0.1 112648   988 pts/1    R+   18:12   0:00 grep -F --color=auto -i swarm

[root@cpm crunchy-postgresql-manager]# psf docker
root      5552  0.1  4.8 394100 24152 pts/1    Sl   18:11   0:00 /usr/bin/docker-current daemon
root      5597  0.0  0.1 112648   992 pts/1    R+   18:13   0:00 grep -F --color=auto -i docker

[root@cpm crunchy-postgresql-manager]# ./sbin/run-swarm
run-swarm-secondary.sh  run-swarm.sh
[root@cpm crunchy-postgresql-manager]# ./sbin/run-swarm.sh
INFO[0000] Listening for HTTP                            addr=10.0.2.15:8000 proto=tcp
ERRO[0000] Get http://:2375/v1.15/info: dial tcp :2375: connection refused
[root@cpm crunchy-postgresql-manager]# INFO[0000] Registering on the discovery service every 20s...  addr=10.0.2.15:2375 discovery=file:///var/cpm/data/swarm_cluster_file
ERRO[0000] not implemented in this discovery service
INFO[0020] Registering on the discovery service every 20s...  addr=10.0.2.15:2375 discovery=file:///var/cpm/data/swarm_cluster_file
ERRO[0020] not implemented in this discovery service

# duplicate lines from swarm INFO[0020] and ERRO[0020] snipped

[root@cpm crunchy-postgresql-manager]# ./run-cpm.sh
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
setting up log dir...
mkdir: cannot create directory ‘/var/cpm/logs’: File exists

# blank lines added for clarity

restarting cpm-web container...
ERRO[0056] HTTP error: No such container: cpm-web        status=404
Error response from daemon: No such container: cpm-web
Error: failed to stop containers: [cpm-web]
ERRO[0056] HTTP error: Container cpm-web not found       status=404
Error response from daemon: Container cpm-web not found
Error: failed to remove containers: [cpm-web]
ERRO[0056] HTTP error: No healthy node available in the cluster  status=500
Error response from daemon: No healthy node available in the cluster

# each restarting container generates the same error messages

restarting cpm-admin container...       # same errors as cpm-web
restarting cpm-task container...        # same errors as cpm-web
restarting cpm-promdash container...    # same errors as cpm-web
restarting cpm-prometheus container...  # same errors as cpm-web
sleeping a bit while cpm-prometheus starts up....
restarting cpm-collect container...     # same errors as cpm-web

# sbipped the swarm generated INFO[0020] abd ERRO[0020] messages from the following

testing containers for DNS resolution....
ping: unknown host cpm-web.crunchy.lab
ping: unknown host cpm-admin.crunchy.lab
ping: unknown host cpm-task.crunchy.lab
ping: unknown host cpm-promdash.crunchy.lab
ping: unknown host cpm-prometheus.crunchy.lab
ping: unknown host cpm-collect.crunchy.lab
ping: unknown host cpm-server.crunchy.lab
jmccormick2001 commented 8 years ago

add this to your docker daemon startup

-H tcp://0.0.0.0:2375

On Thu, May 26, 2016 at 10:25 AM, Dewayne VanHoozer < notifications@github.com> wrote:

I started docker daemon first then swarm. Got a connection error on port

  1. Containers still not resolved.

running as root in $CPMROOT after source cpmenv

[root@cpm crunchy-postgresql-manager]# alias psfalias psf='ps aux | f $@' [root@cpm crunchy-postgresql-manager]# alias falias f='fgrep -i'

[root@cpm crunchy-postgresql-manager]# psf swarm root 5595 0.0 0.1 112648 988 pts/1 R+ 18:12 0:00 grep -F --color=auto -i swarm

[root@cpm crunchy-postgresql-manager]# psf docker root 5552 0.1 4.8 394100 24152 pts/1 Sl 18:11 0:00 /usr/bin/docker-current daemon root 5597 0.0 0.1 112648 992 pts/1 R+ 18:13 0:00 grep -F --color=auto -i docker

[root@cpm crunchy-postgresql-manager]# ./sbin/run-swarm run-swarm-secondary.sh run-swarm.sh [root@cpm crunchy-postgresql-manager]# ./sbin/run-swarm.sh INFO[0000] Listening for HTTP addr=10.0.2.15:8000 proto=tcp ERRO[0000] Get http://:2375/v1.15/info: dial tcp :2375: connection refused [root@cpm crunchy-postgresql-manager]# INFO[0000] Registering on the discovery service every 20s... addr=10.0.2.15:2375 discovery=file:///var/cpm/data/swarm_cluster_file ERRO[0000] not implemented in this discovery service INFO[0020] Registering on the discovery service every 20s... addr=10.0.2.15:2375 discovery=file:///var/cpm/data/swarm_cluster_file ERRO[0020] not implemented in this discovery service

duplicate lines from swarm INFO[0020] and ERRO[0020] snipped

[root@cpm crunchy-postgresql-manager]# ./run-cpm.sh setting up keys dir... mkdir: cannot create directory ‘/var/cpm/keys’: File exists setting up keys dir... mkdir: cannot create directory ‘/var/cpm/keys’: File exists setting up log dir... mkdir: cannot create directory ‘/var/cpm/logs’: File exists

blank lines added for clarity

restarting cpm-web container... ERRO[0056] HTTP error: No such container: cpm-web status=404 Error response from daemon: No such container: cpm-web Error: failed to stop containers: [cpm-web] ERRO[0056] HTTP error: Container cpm-web not found status=404 Error response from daemon: Container cpm-web not found Error: failed to remove containers: [cpm-web] ERRO[0056] HTTP error: No healthy node available in the cluster status=500 Error response from daemon: No healthy node available in the cluster

each restarting container generates the same error messages

restarting cpm-admin container... # same errors as cpm-web restarting cpm-task container... # same errors as cpm-web restarting cpm-promdash container... # same errors as cpm-web restarting cpm-prometheus container... # same errors as cpm-web sleeping a bit while cpm-prometheus starts up.... restarting cpm-collect container... # same errors as cpm-web

sbipped the swarm generated INFO[0020] abd ERRO[0020] messages from the following

testing containers for DNS resolution.... ping: unknown host cpm-web.crunchy.lab ping: unknown host cpm-admin.crunchy.lab ping: unknown host cpm-task.crunchy.lab ping: unknown host cpm-promdash.crunchy.lab ping: unknown host cpm-prometheus.crunchy.lab ping: unknown host cpm-collect.crunchy.lab ping: unknown host cpm-server.crunchy.lab

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221904385

MadBomber commented 8 years ago

I modified the /etc/sysconfig/docker file to use the 0.0.0.0 address. To get a clean slate I exited our of the vagrant vm and halted it. Did a vagrant up and a vagrant ssh to get back in. I'm doing everything as the root user. Docker and swarm seem to be working. I am getting different output with I execute the $CPMROOT/run-cpm.sh script. The pings are still not finding the containers. So I looked at /etc/resolve.conf - It did not have the correct address of the vagrant vm. I executed $CPMROOT/sbin/configure-resolv.sh - it changed things; but the pings of the containers still return host not found errors.

Here is the latest edited script record:

[root@cpm crunchy-postgresql-manager]# ./sbin/configure-resolv.sh
[root@cpm crunchy-postgresql-manager]# cat /etc/resolv.conf
search  crunchy.lab
nameserver  10.0.2.15
nameserver 10.0.2.3

[root@cpm crunchy-postgresql-manager]# cat /etc/resolv.conf.orig
# Generated by NetworkManager
search amer.csc.com crunchy.lab
nameserver 10.0.2.3

[root@cpm crunchy-postgresql-manager]# echo $LOCAL_IP
10.0.2.15

[root@cpm crunchy-postgresql-manager]# psf docker
root       864  0.0  0.2 115244  1432 ?        Ss   13:30   0:00 /bin/sh -c /usr/bin/docker-current daemon $OPTIONS            $DOCKER_STORAGE_OPTIONS            $DOCKER_NETWORK_OPTIONS            $ADD_REGISTRY            $BLOCK_REGISTRY            $INSECURE_REGISTRY            2>&1 | /usr/bin/forward-journald -tag docker
root       869  0.4  7.8 832568 39156 ?        Sl   13:30   0:02 /usr/bin/docker-current daemon --selinux-enabled --bip=172.17.42.1/16 --dns-search=crunchy.lab --dns=10.0.2.15 --dns=192.168.0.1 -H unix:///var/run/docker.sock --label host=10.0.2.15 --label profile=SM -H tcp://0.0.0.0:2375
root       870  0.0  0.3 101728  1816 ?        Sl   13:30   0:00 /usr/bin/forward-journald -tag docker
root      3182  0.0  2.6 180504 13504 ?        Sl   13:32   0:00 docker-proxy -proto tcp -host-ip 10.0.2.15 -host-port 13001 -container-ip 172.17.0.1 -container-port 13001
root      3267  0.0  2.6 180504 13164 ?        Sl   13:32   0:00 docker-proxy -proto tcp -host-ip 10.0.2.15 -host-port 14001 -container-ip 172.17.0.2 -container-port 13001
root      3501  0.0  2.7 180504 13748 ?        Sl   13:32   0:00 docker-proxy -proto tcp -host-ip 10.0.2.15 -host-port 3000 -container-ip 172.17.0.4 -container-port 3000
root      3764  0.0  0.1 112652   992 pts/0    S+   13:39   0:00 grep -F --color=auto -i docker

[root@cpm crunchy-postgresql-manager]# ./sbin/run-swarm.sh
INFO[0000] Listening for HTTP                            addr=10.0.2.15:8000 proto=tcp
INFO[0000] Registered Engine cpm.crunchy.lab at :2375
[root@cpm crunchy-postgresql-manager]# INFO[0000] Registering on the discovery service every 20s...  addr=10.0.2.15:2375 discovery=file:///var/cpm/data/swarm_cluster_file
ERRO[0000] not implemented in this discovery service

[root@cpm crunchy-postgresql-manager]# ./run-cpm.sh
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
setting up keys dir...
mkdir: cannot create directory ‘/var/cpm/keys’: File exists
setting up log dir...
mkdir: cannot create directory ‘/var/cpm/logs’: File exists

# snipped the swarm noise

restarting cpm-web container...
cpm-web
cpm-web
3fde3d9418059f2ae64dd5dbb76a0b1ed438064cda575a163c7a3aa371187b6e
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

restarting cpm-admin container...
cpm-admin
cpm-admin
dcb0ed4eef689f5417664cfde1939515623109484dc4d353665e0602775eb04a
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

restarting cpm-task container...
cpm-task
cpm-task
32e25c782c011d1b5c8866f08dabef2a6be974aef1e5fb065236065526164622
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

restarting cpm-promdash container...
cpm-promdash
cpm-promdash
c5075d90099a473423209f45ce13b31d0ee8d01ab04f2d405e5f9786727e7e11
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

restarting cpm-prometheus container...
cpm-prometheus
cpm-prometheus
495387c751bc8c6cd5c4d3aa26344d355452a7122b631232d4c12a5f46629c76
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

sleeping a bit while cpm-prometheus starts up....
restarting cpm-collect container...
cpm-collect
cpm-collect
327e229e72b6166b9042de9a643ea815ef5207401b99d70a778c4b3aa361e633
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

testing containers for DNS resolution....
ping: unknown host cpm-web.crunchy.lab
ping: unknown host cpm-admin.crunchy.lab
ping: unknown host cpm-task.crunchy.lab
ping: unknown host cpm-promdash.crunchy.lab
ping: unknown host cpm-prometheus.crunchy.lab
ping: unknown host cpm-collect.crunchy.lab
ping: unknown host cpm-server.crunchy.lab
jmccormick2001 commented 8 years ago

I think you are pretty close, the next place to look is for the skybridge container, it should be running and listening on your IP at port 53

docker -H $SWARM_MANAGER_URL start skybridge

skybridge should listen to your Swarm port, and register all the containers with it's skydns.

On Thu, May 26, 2016 at 1:09 PM, Dewayne VanHoozer <notifications@github.com

wrote:

I modified the /etc/sysconfig/docker file to use the 0.0.0.0 address. To get a clean slate I exited our of the vagrant vm and halted it. Did a vagrant up and a vagrant ssh to get back in. I'm doing everything as the root user. Docker and swarm seem to be working. I am getting different output with I execute the $CPMROOT/run-cpm.sh script. The pings are still not finding the containers. So I looked at /etc/resolve.conf - It did not have the correct address of the vagrant vm. I executed $CPMROOT/sbin/configure-resolv.sh - it changed things; but the pings of the containers still return host not found errors.

Here is the latest edited script record:

[root@cpm crunchy-postgresql-manager]# ./sbin/configure-resolv.sh [root@cpm crunchy-postgresql-manager]# cat /etc/resolv.conf search crunchy.lab nameserver 10.0.2.15 nameserver 10.0.2.3

[root@cpm crunchy-postgresql-manager]# cat /etc/resolv.conf.orig# Generated by NetworkManager search amer.csc.com crunchy.lab nameserver 10.0.2.3

[root@cpm crunchy-postgresql-manager]# echo $LOCAL_IP 10.0.2.15

[root@cpm crunchy-postgresql-manager]# psf docker root 864 0.0 0.2 115244 1432 ? Ss 13:30 0:00 /bin/sh -c /usr/bin/docker-current daemon $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY 2>&1 | /usr/bin/forward-journald -tag docker root 869 0.4 7.8 832568 39156 ? Sl 13:30 0:02 /usr/bin/docker-current daemon --selinux-enabled --bip=172.17.42.1/16 --dns-search=crunchy.lab --dns=10.0.2.15 --dns=192.168.0.1 -H unix:///var/run/docker.sock --label host=10.0.2.15 --label profile=SM -H tcp://0.0.0.0:2375 root 870 0.0 0.3 101728 1816 ? Sl 13:30 0:00 /usr/bin/forward-journald -tag docker root 3182 0.0 2.6 180504 13504 ? Sl 13:32 0:00 docker-proxy -proto tcp -host-ip 10.0.2.15 -host-port 13001 -container-ip 172.17.0.1 -container-port 13001 root 3267 0.0 2.6 180504 13164 ? Sl 13:32 0:00 docker-proxy -proto tcp -host-ip 10.0.2.15 -host-port 14001 -container-ip 172.17.0.2 -container-port 13001 root 3501 0.0 2.7 180504 13748 ? Sl 13:32 0:00 docker-proxy -proto tcp -host-ip 10.0.2.15 -host-port 3000 -container-ip 172.17.0.4 -container-port 3000 root 3764 0.0 0.1 112652 992 pts/0 S+ 13:39 0:00 grep -F --color=auto -i docker

[root@cpm crunchy-postgresql-manager]# ./sbin/run-swarm.sh INFO[0000] Listening for HTTP addr=10.0.2.15:8000 proto=tcp INFO[0000] Registered Engine cpm.crunchy.lab at :2375 [root@cpm crunchy-postgresql-manager]# INFO[0000] Registering on the discovery service every 20s... addr=10.0.2.15:2375 discovery=file:///var/cpm/data/swarm_cluster_file ERRO[0000] not implemented in this discovery service

[root@cpm crunchy-postgresql-manager]# ./run-cpm.sh setting up keys dir... mkdir: cannot create directory ‘/var/cpm/keys’: File exists setting up keys dir... mkdir: cannot create directory ‘/var/cpm/keys’: File exists setting up log dir... mkdir: cannot create directory ‘/var/cpm/logs’: File exists

snipped the swarm noise

restarting cpm-web container... cpm-web cpm-web 3fde3d9418059f2ae64dd5dbb76a0b1ed438064cda575a163c7a3aa371187b6e Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

restarting cpm-admin container... cpm-admin cpm-admin dcb0ed4eef689f5417664cfde1939515623109484dc4d353665e0602775eb04a Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

restarting cpm-task container... cpm-task cpm-task 32e25c782c011d1b5c8866f08dabef2a6be974aef1e5fb065236065526164622 Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

restarting cpm-promdash container... cpm-promdash cpm-promdash c5075d90099a473423209f45ce13b31d0ee8d01ab04f2d405e5f9786727e7e11 Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

restarting cpm-prometheus container... cpm-prometheus cpm-prometheus 495387c751bc8c6cd5c4d3aa26344d355452a7122b631232d4c12a5f46629c76 Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

sleeping a bit while cpm-prometheus starts up.... restarting cpm-collect container... cpm-collect cpm-collect 327e229e72b6166b9042de9a643ea815ef5207401b99d70a778c4b3aa361e633 Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.

testing containers for DNS resolution.... ping: unknown host cpm-web.crunchy.lab ping: unknown host cpm-admin.crunchy.lab ping: unknown host cpm-task.crunchy.lab ping: unknown host cpm-promdash.crunchy.lab ping: unknown host cpm-prometheus.crunchy.lab ping: unknown host cpm-collect.crunchy.lab ping: unknown host cpm-server.crunchy.lab

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221949539

MadBomber commented 8 years ago

With skybridge running the pings are still returning unknown hosts.

# On the MacOSX host ...

vagrant halt
vagrant up
vagrant ssh

# On the guest VM ...

sudo su
cd $CPMROOT
source cpmenv
./sbin/run-swarm.sh
./sbin/run-skybridge.sh
./run-cpm.sh

# still getting the unknown host messages on the pings
# here are the sky* processes

root      3219  0.0  0.0  11612   264 ?        Ss   16:17   0:00 /bin/bash -x /var/cpm/bin/start-skybridge.sh
root      3231  0.0  0.5 257412  2848 ?        Sl   16:17   0:00 skydns -addr=0.0.0.0:53 -machines=127.0.0.1:4001 -domain=crunchy.lab. -nameservers=192.168.0.1:53
root      3239  0.1  0.6 117612  3368 ?        Sl   16:17   0:00 skybridge -d crunchy.lab -h tcp://10.0.2.15:8000 -s http://127.0.0.1:4001
jmccormick2001 commented 8 years ago

hmmn, see if you can reach the DNS server....

something like this:

dig @10.1.2.3 cpm.crunchy.lab

use your IP address instead of 10.1.2.3

also 'netstat -na | grep LISTEN | grep 53' should show the port, would be good to see that output.

On Thu, May 26, 2016 at 3:34 PM, Dewayne VanHoozer <notifications@github.com

wrote:

With skybridge running the pings are still returning unknown hosts.

On the MacOSX host ...

vagrant halt vagrant up vagrant ssh

On the guest VM ...

sudo sucd $CPMROOTsource cpmenv ./sbin/run-swarm.sh ./sbin/run-skybridge.sh ./run-cpm.sh

still getting the unknown host messages on the pings# here are the sky* processes

root 3219 0.0 0.0 11612 264 ? Ss 16:17 0:00 /bin/bash -x /var/cpm/bin/start-skybridge.sh root 3231 0.0 0.5 257412 2848 ? Sl 16:17 0:00 skydns -addr=0.0.0.0:53 -machines=127.0.0.1:4001 -domain=crunchy.lab. -nameservers=192.168.0.1:53 root 3239 0.1 0.6 117612 3368 ? Sl 16:17 0:00 skybridge -d crunchy.lab -h tcp://10.0.2.15:8000 -s http://127.0.0.1:4001

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-221986611

MadBomber commented 8 years ago

This is strange.

[root@cpm crunchy-postgresql-manager]# source cpmenv
[root@cpm crunchy-postgresql-manager]# echo $LOCAL_IP
10.0.2.15
[root@cpm crunchy-postgresql-manager]# dig @10.0.2.15 cpm.crunchy.lab

; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> @10.0.2.15 cpm.crunchy.lab
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27950
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;cpm.crunchy.lab.   IN  A

;; AUTHORITY SECTION:
crunchy.lab.    60  IN  SOA ns.dns.crunchy.lab. hostmaster.skydns.local. 1464300000 28800 7200 604800 60

;; Query time: 9 msec
;; SERVER: 10.0.2.15#53(10.0.2.15)
;; WHEN: Thu May 26 18:04:56 EDT 2016
;; MSG SIZE  rcvd: 99

# netstat is showing nothing on port 53
[root@cpm crunchy-postgresql-manager]# netstat -na | f listen
tcp        0      0 10.0.2.15:13001         0.0.0.0:*               LISTEN
tcp        0      0 10.0.2.15:14001         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 10.0.2.15:3000          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 10.0.2.15:8000          0.0.0.0:*               LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 ::1:25                  :::*                    LISTEN
tcp6       0      0 :::2375                 :::*                    LISTEN

# The 'ps aux' shows that skydns was started on port 53
# Is the nameserver address correct?

[root@cpm crunchy-postgresql-manager]# psf sky
root      3219  0.0  0.0  11612    48 ?        Ss   16:17   0:00 /bin/bash -x /var/cpm/bin/start-skybridge.sh
root      3231  0.0  0.5 257796  2880 ?        Sl   16:17   0:00 skydns -addr=0.0.0.0:53 -machines=127.0.0.1:4001 -domain=crunchy.lab. -nameservers=192.168.0.1:53
root      3239  0.1  0.3 117612  1692 ?        Sl   16:17   0:08 skybridge -d crunchy.lab -h tcp://10.0.2.15:8000 -s http://127.0.0.1:4001
root      4912  0.0  0.1 112648   988 pts/1    R+   18:11   0:00 grep -F --color=auto -i sky
jmccormick2001 commented 8 years ago

you might try setting selinux to permissive mode and also make sure to turn off the firewall if it running

I've seen issues with those before.

On Thu, May 26, 2016 at 5:16 PM, Dewayne VanHoozer <notifications@github.com

wrote:

This is strange.

[root@cpm crunchy-postgresql-manager]# source cpmenv [root@cpm crunchy-postgresql-manager]# echo $LOCAL_IP 10.0.2.15 [root@cpm crunchy-postgresql-manager]# dig @10.0.2.15 cpm.crunchy.lab ; <<>> DiG 9.9.4-RedHat-9.9.4-29.el7_2.3 <<>> @10.0.2.15 cpm.crunchy.lab; (1 server found);; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27950;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION:;cpm.crunchy.lab. IN A ;; AUTHORITY SECTION: crunchy.lab. 60 IN SOA ns.dns.crunchy.lab. hostmaster.skydns.local. 1464300000 28800 7200 604800 60 ;; Query time: 9 msec;; SERVER: 10.0.2.15#53(10.0.2.15);; WHEN: Thu May 26 18:04:56 EDT 2016;; MSG SIZE rcvd: 99

netstat is showing nothing on port 53

[root@cpm crunchy-postgresql-manager]# netstat -na | f listen tcp 0 0 10.0.2.15:13001 0.0.0.0:* LISTEN tcp 0 0 10.0.2.15:14001 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 10.0.2.15:3000 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 10.0.2.15:8000 0.0.0.0:* LISTEN tcp6 0 0 :::22 :::* LISTEN tcp6 0 0 ::1:25 :::* LISTEN tcp6 0 0 :::2375 :::* LISTEN

The 'ps aux' shows that skydns was started on port 53# Is the nameserver address correct?

[root@cpm crunchy-postgresql-manager]# psf sky root 3219 0.0 0.0 11612 48 ? Ss 16:17 0:00 /bin/bash -x /var/cpm/bin/start-skybridge.sh root 3231 0.0 0.5 257796 2880 ? Sl 16:17 0:00 skydns -addr=0.0.0.0:53 -machines=127.0.0.1:4001 -domain=crunchy.lab. -nameservers=192.168.0.1:53 root 3239 0.1 0.3 117612 1692 ? Sl 16:17 0:08 skybridge -d crunchy.lab -h tcp://10.0.2.15:8000 -s http://127.0.0.1:4001 root 4912 0.0 0.1 112648 988 pts/1 R+ 18:11 0:00 grep -F --color=auto -i sky

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/CrunchyData/crunchy-postgresql-manager/issues/121#issuecomment-222010910

MadBomber commented 8 years ago

no joy in mudville ...

My test sequence:

# On Host Mac
vagrant halt
vagrant up
vagrant ssh

# On guest centos-7
sudo su
cd $CPMROOT

getenforce # showed enforcing
setenforce 0
getenforce # showed permissive

firewall-cmd --state # showed not running

source cpmenv
./sbin/run-swarm.sh
./sbin/run-skybridge.sh
./run-cpm.sh

# nojoy ... pings still return unknown host for the containers

Using the 'strings' command on the /var/cpm/data/etcd/wal/00.wal file I was able to determine the IP addresses associated with these containers. There were no other cpm related entries.

172.17.0.2  cpm-web.crunchy.lab
172.17.0.3  cpm-admin.crunchy.lab
172.17.0.4  cpm-task.crunchy.lab
172.17.0.5  cpm-promdash.crunchy.lab
172.17.0.6  cpm-collect.crunchy.lab

Each of these IP address was successfully pinged.

At this point I am going to claim that the initial build process of the vagrant up command failed because of unspecified network lockdowns on my office network. I am going to blow away this VM and rebuild it from step 1 at home.

Lets close this issue.

jmccormick2001 commented 8 years ago

will do