openshift / origin

Conformance test suite for OpenShift
http://www.openshift.org
Apache License 2.0
8.48k stars 4.7k forks source link

oc cluster up should check and error/warn if ports 80/443 already in use #10665

Open GrahamDumpleton opened 8 years ago

GrahamDumpleton commented 8 years ago

If port 80 is already in use by a web server on the host when running oc cluster up, there is no error or warning when starting up. What happens is that the router pod keeps failing instead and so although the web interface works and can deploy applications, you can't access the deployed applications. If you use docker logs on dead router containers they show no log messages at all.

The oc cluster up command should perhaps do a TCP connection to both 80 and 443 when run and warn that the ports are already in use and that routing may not work for applications.

So what normally expect to see from docker ps is:

CONTAINER ID        IMAGE                                                                                                          COMMAND                  CREATED             STATUS              PORTS                                                              NAMES
33a27c112089        172.30.42.184:5000/myproject/wagtail@sha256:5df865eb590ec71fb9a45b9a98b7a8863e65bed3cbf0bf8fd70d6a5d1ca74984   "container-entrypoint"   15 minutes ago      Up 15 minutes                                                                          k8s_wagtail.a0b2fe21_wagtail-2-3l4ms_myproject_e8615470-6b38-11e6-9a94-8edd4be1d78f_f2f9920b
55130194ce03        172.30.42.184:5000/myproject/hello@sha256:4488e828e3cf7ba562dd213838fa0a527bd99b08a93447a5711becfcf094247d     "container-entrypoint"   15 minutes ago      Up 15 minutes                                                                          k8s_hello.7407bde5_hello-1-2bfkb_myproject_c7ba53cb-6b36-11e6-9a94-8edd4be1d78f_441f972b
0a90f8b8b932        openshift/origin-haproxy-router:v1.3.0-alpha.3                                                                 "/usr/bin/openshift-r"   15 minutes ago      Up 15 minutes                                                                          k8s_router.b2493d1b_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_4dd0cbb3
7797adc309f7        openshift/origin-docker-registry:v1.3.0-alpha.3                                                                "/bin/sh -c 'DOCKER_R"   15 minutes ago      Up 15 minutes                                                                          k8s_registry.37080643_docker-registry-1-ldnfy_default_f59e60f5-6b35-11e6-9a94-8edd4be1d78f_187bf85b
122ce3815535        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   15 minutes ago      Up 15 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:1936->1936/tcp   k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_44567a6e
3a577c8b5af9        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   15 minutes ago      Up 15 minutes                                                                          k8s_POD.f98c0130_hello-1-2bfkb_myproject_c7ba53cb-6b36-11e6-9a94-8edd4be1d78f_fc361b0b
c7ded9a6ee05        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   15 minutes ago      Up 15 minutes                                                                          k8s_POD.f98c0130_wagtail-2-3l4ms_myproject_e8615470-6b38-11e6-9a94-8edd4be1d78f_b704106c
8531bddd7ce7        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   15 minutes ago      Up 15 minutes                                                                          k8s_POD.e4a40125_docker-registry-1-ldnfy_default_f59e60f5-6b35-11e6-9a94-8edd4be1d78f_bd2782e1
eaceae32fe3f        openshift/origin:v1.3.0-alpha.3                                                                                "/usr/bin/openshift s"   15 minutes ago      Up 15 minutes                                                                          origin

If start up Apache httpd server on port 80 and run oc cluster up, then output all looks normal.

- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.3.0-alpha.3 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using Docker shared volumes for OpenShift volumes
-- Checking Docker version ... OK
-- Creating host directories ... OK
-- Finding server IP ...
   Using 192.168.178.53 as the server IP
-- Starting OpenShift container ...
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ...
   Already on project "myproject" on server "https://192.168.178.53:8443".
-- Server Information ...
   OpenShift server started.
   The server is accessible via web console at:
       https://192.168.178.53:8443

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

The router pod doesn't show as running in docker ps.

CONTAINER ID        IMAGE                                                                                                          COMMAND                  CREATED             STATUS              PORTS               NAMES
476eb0da098b        172.30.42.184:5000/myproject/wagtail@sha256:5df865eb590ec71fb9a45b9a98b7a8863e65bed3cbf0bf8fd70d6a5d1ca74984   "container-entrypoint"   31 seconds ago      Up 30 seconds                           k8s_wagtail.a0b2fe21_wagtail-2-3l4ms_myproject_e8615470-6b38-11e6-9a94-8edd4be1d78f_57c70440
24ff28b1005e        172.30.42.184:5000/myproject/hello@sha256:4488e828e3cf7ba562dd213838fa0a527bd99b08a93447a5711becfcf094247d     "container-entrypoint"   32 seconds ago      Up 32 seconds                           k8s_hello.7407bde5_hello-1-2bfkb_myproject_c7ba53cb-6b36-11e6-9a94-8edd4be1d78f_8448fa36
fc09d8f5251c        openshift/origin-docker-registry:v1.3.0-alpha.3                                                                "/bin/sh -c 'DOCKER_R"   46 seconds ago      Up 46 seconds                           k8s_registry.37080643_docker-registry-1-ldnfy_default_f59e60f5-6b35-11e6-9a94-8edd4be1d78f_b042101c
a258fb54eb4f        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   47 seconds ago      Up 46 seconds                           k8s_POD.f98c0130_wagtail-2-3l4ms_myproject_e8615470-6b38-11e6-9a94-8edd4be1d78f_90e35c2b
8d02c9a742d7        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   47 seconds ago      Up 46 seconds                           k8s_POD.e4a40125_docker-registry-1-ldnfy_default_f59e60f5-6b35-11e6-9a94-8edd4be1d78f_994a0097
77d7d0f706c8        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   47 seconds ago      Up 46 seconds                           k8s_POD.f98c0130_hello-1-2bfkb_myproject_c7ba53cb-6b36-11e6-9a94-8edd4be1d78f_70fa4e88
aff1d797b617        openshift/origin:v1.3.0-alpha.3                                                                                "/usr/bin/openshift s"   58 seconds ago      Up 57 seconds                           origin

If you run docker ps -a you can though see many failed router pods.

CONTAINER ID        IMAGE                                                                                                          COMMAND                  CREATED              STATUS              PORTS               NAMES
6160dacd9ae2        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   10 seconds ago       Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_55b57733
6e7c0140583d        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   24 seconds ago       Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_18dd0ab7
1750c23d341e        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   38 seconds ago       Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_424b467e
a8567a2edc80        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   51 seconds ago       Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_cbb66b1a
c157a4d62e7a        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   52 seconds ago       Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_0908136f
4811d6bb8506        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   58 seconds ago       Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_24e74743
589d8af5cf32        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   About a minute ago   Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_b635041d
3d4af8c45b8e        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   About a minute ago   Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_524edd4a
7418063b633b        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   About a minute ago   Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_bfcc582a
cdb3377f69d6        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   About a minute ago   Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_a1a8fbb0
b317f733e393        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   2 minutes ago        Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_1a7804f2
c43af56738b0        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   2 minutes ago        Created                                 k8s_POD.561a4a31_router-1-dz2wk_default_f57c2c14-6b35-11e6-9a94-8edd4be1d78f_1a40c13f
476eb0da098b        172.30.42.184:5000/myproject/wagtail@sha256:5df865eb590ec71fb9a45b9a98b7a8863e65bed3cbf0bf8fd70d6a5d1ca74984   "container-entrypoint"   2 minutes ago        Up 2 minutes                            k8s_wagtail.a0b2fe21_wagtail-2-3l4ms_myproject_e8615470-6b38-11e6-9a94-8edd4be1d78f_57c70440
24ff28b1005e        172.30.42.184:5000/myproject/hello@sha256:4488e828e3cf7ba562dd213838fa0a527bd99b08a93447a5711becfcf094247d     "container-entrypoint"   2 minutes ago        Up 2 minutes                            k8s_hello.7407bde5_hello-1-2bfkb_myproject_c7ba53cb-6b36-11e6-9a94-8edd4be1d78f_8448fa36
fc09d8f5251c        openshift/origin-docker-registry:v1.3.0-alpha.3                                                                "/bin/sh -c 'DOCKER_R"   2 minutes ago        Up 2 minutes                            k8s_registry.37080643_docker-registry-1-ldnfy_default_f59e60f5-6b35-11e6-9a94-8edd4be1d78f_b042101c
a258fb54eb4f        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   2 minutes ago        Up 2 minutes                            k8s_POD.f98c0130_wagtail-2-3l4ms_myproject_e8615470-6b38-11e6-9a94-8edd4be1d78f_90e35c2b
8d02c9a742d7        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   2 minutes ago        Up 2 minutes                            k8s_POD.e4a40125_docker-registry-1-ldnfy_default_f59e60f5-6b35-11e6-9a94-8edd4be1d78f_994a0097
77d7d0f706c8        openshift/origin-pod:v1.3.0-alpha.3                                                                            "/pod"                   2 minutes ago        Up 2 minutes                            k8s_POD.f98c0130_hello-1-2bfkb_myproject_c7ba53cb-6b36-11e6-9a94-8edd4be1d78f_70fa4e88
aff1d797b617        openshift/origin:v1.3.0-alpha.3                                                                                "/usr/bin/openshift s"   2 minutes ago        Up 2 minutes                            origin

As said above, using docker logs on any of the router pods shows no log messages at all, so would have to guess issue is that something is running on port 80.

akram commented 8 years ago

In my case, I wanted to run the OpenShift router on alternative port: 1080 and 10443. And I ended up in this case

-- Checking for available ports ... FAIL
   Error: a port needed by OpenShift is not available
   Caused By:
     Error: ports in use: [443]

So maybe the check has been implemented in the meantime or it was already existing. In both case, it prevents from running the router on other ports. Should we add an option? o

akram commented 8 years ago

@GrahamDumpleton apparently the check is done here https://github.com/openshift/origin/blob/d89bdf3fe4c45562e7ed190330898c5804cd4456/pkg/bootstrap/docker/up.go#L515

@pweil- @csrwng should we mark this issue as closed? I want a parameter to change the default OpenShift Base port also. Should I open a new issue?

csrwng commented 8 years ago

@akram this issue is still valid ... the problem is that we're currently only checking the docker host ... however when starting up on a Mac, we will eventually expose the port to the Mac's network interface through something like docker run -p 80:80 blah and that's the part that fails.

akram commented 8 years ago

@csrwng thanks, indeed, I saw it working when running on RHEL and not in docker-machine.

akram commented 8 years ago

I created #10717 to allow one to skip this verification once we got it fully implemented

coreydaley commented 7 years ago

Pull request #11600 was merged which does correct the checks to be warnings instead of failing, but does not yet correct this issue. Another pull request will be opened for this issue.

coreydaley commented 7 years ago

@akram Yes, you would need to open a new issue to add a new parameter to change the default OpenShift base port.

coreydaley commented 7 years ago

@csrwng Currently, the only place I see a socat tunnel being configured is in the helper_unix.go, and it is for port 8443. Could you point me towards where socat is being used to forward ports 80 & 443 ?

csrwng commented 7 years ago

@coreydaley the problem is not the port socat opens. It's the port that docker opens when running the router. It essentially exposes ports on the mac with the -p flag. However, we only do that when running socat.

frobware commented 6 years ago

As a me-too, I ran into this issue today. What was worse for me is I had something else restarting a service on port 80 and sometimes oc cluster up would fail and sometime not, largely dependent on timing.

openshift-bot commented 6 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

GrahamDumpleton commented 6 years ago

/lifecycle frozen

adambkaplan commented 6 years ago

This is still an issue after the 3.10 update using the component-based install. The router component should do the following:

  1. Check if it is using host port forwarding (currently only done on macOS)
  2. If so, check that the ports for the router are not in use.
  3. If ports are in use, fail the component installation.
adambkaplan commented 6 years ago

@openshift/sig-master in your wheelhouse since this is related to cluster up

mfojtik commented 6 years ago

@adambkaplan the router component installer is owned by the @openshift/sig-networking team ;-)

/cc @ironcladlou

ironcladlou commented 6 years ago

To make sure I understand: this issue pertains only to oc cluster up against a remote docker daemon (e.g. MacOS use cases), correct?

GrahamDumpleton commented 6 years ago

Don't know about remote docker daemon. If you mean where docker runs in a VM, such as on macOS and Windows, then yes.

danwinship commented 5 years ago

/remove-sig networking /sig network-edge