ansibleapp-etherpad not working with oc cluster up

[munchee13]

I have tried this with both the upstream "oc cluster up" with Origin 1.5 and also with the enterprise version of OpenShift 3.4.1.2. Here is my output from the latter.

You will note the following line as the culprit: error: dial tcp 127.0.0.1:8443: getsockopt: connection refused

Here is more information about my host environment:

• No Firewall running at all
• Using Docker for Mac Version 1.13.1-rc1-beta40
• Using macOS Sierra 10.12.4 Beta

For comparison, using the Origin VM environment instead of "oc cluster up" works with no connection issues. However, the official Red Hat CDK and the latest enterprise oc command both use "oc cluster up" under the covers and are the likely (and eventually preferred) means we expect partners to do their development with the service broker.

Let me know if I can provide anything else.

[dymurray]

@munchee13 It looks like the problem is your openshift_target IP address points to localhost. The issue is that the container needs to authenticate against the OC cluster which is not running on localhost in retrospect to the container. Change the openshift_target IP to the IP address of your host assigned on the network.

[munchee13]

localhost is where it is running — using the laptop IP does not change the behavior — did “oc cluster up” work for you?

docker run -e "OPENSHIFT_TARGET=https://192.168.1.13:8443" -e "OPENSHIFT_USER=developer" -e "OPENSHIFT_PASS=developer" ansibleapp/etherpad-ansibleapp provision error: dial tcp 192.168.1.13:8443: getsockopt: connection refused

PLAY [Deploy playbook-etherpad to openshift] **

TASK [/usr/local/ansible/roles/playbook-etherpad-openshift : oso_service] ** fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Error switching to project playbook-etherpad", "stderr": "error: Missing or incomplete configuration info. Please login or point to an existing, complete config file:\n\n 1. Via the command-line flag --config\n 2. Via the KUBECONFIG environment variable\n 3. In your home directory as ~/.kube/config\n\nTo view or setup config directly use the 'config' command.\nSee 'oc project -h' for help and examples.\n", "stdout": "", "stdout_lines": []} to retry, use: --limit @/ansibleapp/actions/provision.retry

PLAY RECAP ***** localhost : ok=0 changed=0 unreachable=0 failed=1

On Feb 3, 2017, at 6:19 PM, Dylan Murray notifications@github.com wrote:

@munchee13 https://github.com/munchee13 It looks like the problem is your openshift_target IP address points to localhost. The issue is that the container needs to authenticate against the OC cluster which is not running on localhost in retrospect to the container. Change the openshift_target IP to the IP address of your host assigned on the network.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fusor/ansibleapp-library/issues/5#issuecomment-277389755, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxhZcflALrjE1cOQfYiOZJYv3SfOuXIks5rY7YbgaJpZM4L26Tt.

[munchee13]

FWIW, the following works:

oc login https://127.0.0.1:8443 -u developer Logged into "https://127.0.0.1:8443" as "developer" using existing credentials.

You have one project on this server: "myproject"

Using project "myproject".

[dymurray]

@munchee13 So this is what I had to do to get it working by using oc cluster up. I ran oc cluster up and my server IP address was an address on my libvirt host (192.168.121.*). I was then able to use this as a target host IP and it worked great. I am unsure what changes when the cluster server runs on localhost.

I would be curious if you can run an interactive bash shell from the container and try to run an 'oc login' against the cluster somehow because that is the only part of the code that is really failing and once I solved the target host IP being accessible from a new container being spawned it solved my issue. I will continue testing with oc cluster up to see how I can help.

[munchee13]

Thanks… I’m using cluster up with the native docker on my laptop — no VM is generated, so libvirt nor any other hypervisor is in use.

On Feb 3, 2017, at 7:02 PM, Dylan Murray notifications@github.com wrote:

@munchee13 https://github.com/munchee13 So this is what I had to do to get it working by using oc cluster up. I ran oc cluster up and my server IP address was an address on my libvirt host (192.168.121.*). I was then able to use this as a target host IP and it worked great. I am unsure what changes when the cluster server runs on localhost.

I would be curious if you can run an interactive bash shell from the container and try to run an 'oc login' against the cluster somehow because that is the only part of the code that is really failing and once I solved the target host IP being accessible from a new container being spawned it solved my issue. I will continue testing with oc cluster up to see how I can help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fusor/ansibleapp-library/issues/5#issuecomment-277396348, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxhZaHnkiVHx1XoSplyTOopEcrEio9Iks5rY8ADgaJpZM4L26Tt.

[dymurray]

Ah... yes. Try using the host IP on the docker network. I know that using localhost as the target IP will not work in this scenario as it will point to localhost of the container.

[munchee13]

That’s the 192.168.1.13 — it doesn’t work per the previous comment. Is there another IP to which you are referring?

On Feb 3, 2017, at 7:09 PM, Dylan Murray <notifications@github.com mailto:notifications@github.com> wrote:

Ah... yes. Try using the host IP on the docker network. I know that using localhost as the target IP will not work in this scenario as it will point to localhost of the container.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fusor/ansibleapp-library/issues/5#issuecomment-277397334, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxhZQAKjPGo1Q25VHVSh0CeS8X2-rQEks5rY8GxgaJpZM4L26Tt.

[dymurray]

[vagrant@cap openshift-origin-client-tools-v1.3.1-dad658de7465ba8a234a4fb40b5b446a45a4cee1-linux-64bit]$ ifconfig br-c25c065473a2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.18.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet6 fe80::42:dcff:fe66:9fa8 prefixlen 64 scopeid 0x20 ether 02:42:dc:66:9f:a8 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet6 fe80::42:4aff:fe78:d1ff prefixlen 64 scopeid 0x20 ether 02:42:4a:78:d1:ff txqueuelen 0 (Ethernet) RX packets 1869866 bytes 227075854 (216.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0172.17.0.1 TX packets 2250476 bytes 2899182862 (2.7 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I have an IP on the docker subnet (which for the host is usually the same as mine) and its 172.17.0.1.

I'm unsure of your networking setup though.

[munchee13]

There is no docker0 network when you use native Docker for Mac -- I think that is the difference. Here is my entire ifconfig. Remember, I'm NOT using a VM.

[dymurray]

@munchee13 I see. I am going to try and replicate your setup and will report back to you. Thanks for the note.

[eriknelson]

@munchee13 Unfortunately I don't have a mac to try this on, but I wonder what the network setup looks up from a container on mac? What are the outputs from these commands? I'm expecting it to look pretty standard with the containers on 172.17.0.0/16, but it's worth seeing if they've got something strange going on.

docker run --entrypoint=ifconfig ansibleapp/etherpad-ansibleapp

docker run --entrypoint=route ansibleapp/etherpad-ansibleapp

Are you running anything in a Virtualbox VM? I think docker for mac runs the engine inside a VM and the mac client talks to it via a vboxnet.

Does "OPENSHIFT_TARGET=https://10.2.2.1:8443" work?

[eriknelson]

Dug out my old MacBook and managed to stand this up on Sierra. docker-machine ip openshift should return the ip of the engine's box that the metacontainer can talk to.

docker run -e "OPENSHIFT_TARGET=$(docker-machine ip openshift):8443" -e "OPENSHIFT_USER=developer" -e "OPENSHIFT_PASS=developer" ansibleapp/etherpad-ansibleapp provision

That should at least get you talking to the cluster. There are a couple other differences I saw with this setup compared to our CDK vagrant env that blocked me from having etherpad successfully launch. The project was missing, so activation failed. After a manual oc new-project playbook-etherpad, I got a bunch of filesystem permission problems in the etherpad pod.

We'll troubleshoot these and report back.

[munchee13]

Guys... this is not using Docker Machine. This is using Native Docker for Mac. The docker-machine is their old solution which did use a VM.

[eriknelson]

@munchee13 Sorry, I was misdirected by your vboxnet in the ifconfig. Docker for Mac indeed still uses a Hyperkit VM as opposed to the old Vbox. Sounds like a very lightweight, embeddable hypervisor. I don't know anything about this, but I'll see if I can get things working now that I have a mac environment to play with.

[munchee13]

Let me offer more clarity, so the problem can be addressed and not the symptom. There are 4 primary means (to date) that an a developer will work with OpenShift on their laptop for individual development and are use cases we should consider with the developer experience:

1. Using oc cluster up with a native docker runtime. This implies that docker is running directly on the host with no VM. This is where this particular issue is hit. The docker runtime is on localhost and that is also where the OpenShift master is deployed and running.

2. Using oc cluster up with a docker-machine runtime implementation. This implies that there is a VM layer instantiated first with OpenShift deployed within. This is what you are experiencing if you do NOT have a native docker runtime as linked above.

3. Using minishift or the official Red Hat CDK. The former is the upstream for the latter. Today minishift implements oc cluster up under the covers using a docker-machine implementation as defined in number 2.

4. Using the Origin All-in-One VM. This method will likely be deprecated eventually in favor of one of the above methods.

The only one where there is a communication issue is with the first one. However, there are two other issues to consider with items 1 through 3 as well. One is the expectation that the project is already created and has a specific name. This should be a passable parameter, I think, and if it is not there, then create the default automatically for me. I opened an issue here. The second is that the container examples are using root. If we want to continue using root containers, then we likely need to provide instructions to users on how to allow root containers to work in their cluster. You do not hit this issue with the Origin VM use case. I opened an issue on this here.

I hope all of this helps -- I realize it's early, but I think this could be the most important ecosystem feature for OpenShift this year, and I'm excited to see it implemented. Keep up the awesome work!

[eriknelson]

Thank you @munchee13, that was informative. It would be good to test against these to make sure we're covering our bases moving forwards. Will continue to look into this and report back.

[munchee13]

👍🏻

Cheers, Chris

On Feb 4, 2017, at 8:54 AM, Erik Nelson notifications@github.com wrote:

Thank you @munchee13, that was informative. It would be good to test against 1-3 to make sure we're covering our bases moving forwards. Will continue to look into this and report back.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[eriknelson]

@munchee13 I think the problem here is the origin cluster is listening on 127.0.0.1:8443 when using Docker for Mac. Other containers (the meta container), can't see it from inside their context.

# Inside running ansibleapp/etherpad-ansibleapp container
[root@b10a6ef3e520 /]# nmap 10.192.219.224

Starting Nmap 6.40 ( http://nmap.org ) at 2017-02-06 13:52 UTC
Nmap scan report for 10.192.219.224
Host is up (0.0021s latency).
Not shown: 500 closed ports, 498 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 15.89 seconds

---

# nmap localhost

Starting Nmap 7.40 ( https://nmap.org ) at 2017-02-06 08:55 EST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0010s latency).
Other addresses for localhost (not scanned): ::1 fe80::1
Not shown: 749 closed ports, 248 filtered ports
PORT     STATE SERVICE
80/tcp   open  http
443/tcp  open  https
8443/tcp open  https-alt

Nmap done: 1 IP address (1 host up) scanned in 4.34 seconds
{> ~ <}
# lsof -Pn -i4 | grep 8443
socat     4999 nelsk    5u  IPv4 0x6b5320f8c2d8dcbb      0t0  TCP 127.0.0.1:8443 (LISTEN)

Is there a way to run an oc cluster up and listen on a host other than 127.0.0.1?

[munchee13]

There is a public-hostname flag that likely does it, but that hostname would need to resolve to something other than 127.0.0.1 which may not be the case. What is weird is that anything else that I deploy to the cluster works -- it's just these examples.

On Feb 6, 2017, at 9:59 AM, Erik Nelson notifications@github.com wrote:

@munchee13 I think the problem here is the origin cluster is listening on 127.0.0.1:8443 when using Docker for Mac. Other containers (the meta container), can't see it from inside their context.

Inside running ansibleapp/etherpad-ansibleapp container

[root@b10a6ef3e520 /]# nmap 10.192.219.224

Starting Nmap 6.40 ( http://nmap.org ) at 2017-02-06 13:52 UTC Nmap scan report for 10.192.219.224 Host is up (0.0021s latency). Not shown: 500 closed ports, 498 filtered ports PORT STATE SERVICE 80/tcp open http 443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 15.89 seconds

---

nmap localhost

Starting Nmap 7.40 ( https://nmap.org ) at 2017-02-06 08:55 EST Nmap scan report for localhost (127.0.0.1) Host is up (0.0010s latency). Other addresses for localhost (not scanned): ::1 fe80::1 Not shown: 749 closed ports, 248 filtered ports PORT STATE SERVICE 80/tcp open http 443/tcp open https 8443/tcp open https-alt

Nmap done: 1 IP address (1 host up) scanned in 4.34 seconds {> ~ <}

lsof -Pn -i4 | grep 8443

socat 4999 nelsk 5u IPv4 0x6b5320f8c2d8dcbb 0t0 TCP 127.0.0.1:8443 (LISTEN) Is there a way to run an oc cluster up and listen on a host other than 127.0.0.1?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.