csrwng commented 6 years ago

Docker for Mac CE 17.12.0 introduced an internal proxy for Docker that prevents pushes into the local OpenShift registry when running a cluster with cluster up

You also get a warning when starting cluster up like:

WARNING: An HTTP proxy (docker.for.mac.http.internal:3128) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: An HTTPS proxy (docker.for.mac.http.internal:3129) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list.
   172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

For now, the workaround is to downgrade Docker to a version prior to 17.12.0. You can find download links to previous releases in their release note page: https://docs.docker.com/docker-for-mac/release-notes/

Open Issues: https://github.com/docker/for-mac/issues/2506 https://github.com/docker/for-mac/issues/2470 https://github.com/docker/for-mac/issues/2467 https://github.com/docker/for-mac/issues/2386

csrwng commented 6 years ago

@mfojtik @deads2k /cc @mpbarrett

mfojtik commented 6 years ago

I would hold on fixing this until @deads2k finish the internal refactoring of cluster up.

riuvshin commented 6 years ago

same here, is there any other workaround for this? I don't want to downgrade... adding 172.30.1.1 to no_proxy in docker prefs does not help docker Version 18.02.0-ce-mac53 (22617)

alberttwong commented 6 years ago

I haven't be able to find a work around.. No proxy doesn't seem to work.

mikz commented 6 years ago

Docker Community Edition 18.03.0-ce-mac59 2018-03-26 Fix for the HTTP/S transparent proxy when using “localhost” names (e.g. host.docker.internal).

This does not fix the issue. So don't upgrade.

cmcconnell1 commented 6 years ago

Confirming that I'm still seeing this with 2018-04-30 18.03.1-ce-mac65

Valien commented 6 years ago

So decided to try this again and still broken with docker version 18.03.1-ce. At least it's nice to know it's a verified issue/bug. Bummed there is no resolution besides using an older version of docker though.

cmcconnell1 commented 6 years ago

Apologies, I previously automated workarounds for our team shortly after my posts above and then I forgot about this issue. I'm new to Minishift/Openshift (coming from deis workflow && looking at OS Origin to provide a solid PaaS solution), and not sure if this would work for everyone.

TL;DR: Here is my initial POC (with docker for mac minishift workaround) for a local minishift cluster: minishift-local-poc Its automated and allows our distributed teams to quickly config and deploy our platform within a few minutes.

Details

For our use case, we're working locally with minishift, pulling from remote private AWS ECR repos, tagging and pushing our images to the remote (Openshift / Minishift registry--AFAIK, minishift still considers this to be a remote) docker registry.
I had found another blog post from last year where they needed to use secrets else they were getting the permission denied errors (as was I) so set those up, and automated that process as well. I'm not sure if that's the correct "Openshift/Minishift way," but it works for us so I went with it.
My solution configures a minishift cluster with requisite configs such that future starts can just be minishift start and it will use the default profile configurations specified in the script. Then the parent script spawns child script/processes, which each configure requisite apps for our platform in separate minishift project/namespaces--with each app pulling/tagging their requisite docker images from private AWS ECR repo and pushing into the Minishift docker registry--i.e.:
```
docker push 172.30.1.1:5000/myapp-local/myapp
```
Another bit of advice from my experience was that the virtualbox driver just worked--('seem to recall some other drivers missing plugins, configurations, etc.) so went with VB driver and that = profit for me. JIC, my script will install the requisite minishift plugins, enable them, and also set admin creds--this should only be used in local/dev env's as it is not secure. So it will configure the driver to be virtualbox as the default, but you can easily change that as you wish.
Notes I'm currently Not running local docker, but using Minishift docker context--will that work for you? This parent script sets the Docker context to the Minishift Registry and thus, I'm not even running Docker at all on my OSX laptop/workstation
- Note: see the above noted git repo for the minishift deployment script which pushes, to Minishift docker registry without any local docker for mac running.
```
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
```
  AIR, you Can, have local docker running with this script without issues too, since the script will set the requisite context.
```
DOCKER CONTEXT
# switch to minishift context
eval $(minishift docker-env)
#eval $(minikube docker-env) # if we were using minikube
```

env | grep -i docker DOCKER_HOST=tcp://192.168.99.100:2376 DOCKER_TLS_VERIFY=1 DOCKER_CERT_PATH=/Users/cmcc/.minishift/certs

switch back to your machines context do:

eval $(docker-machine env -u)



I hope this helps others and would love to hear how folks are automating their docker images in Openshift Origin in H/A production environments, as I'm currently rolling my own scripts ATM to facilitate app creation, deployment, rollback, etc.  
I'm curious if the masses would recommend embracing using Openshift templates?  This seems very cool to be able to export any/all requisite NS/Project artifacts and codify/promote/manage them, etc.

-cmcc

mikz commented 6 years ago

@cmcconnell1 I just settled on oc cluster up --create-machine.

patelbharat commented 6 years ago

I am seeing this issue on docker version 18.06.1-ce-mac73 as well. Had to downgrade to 17.09 as a work around

fg78nc commented 6 years ago

@patelbharat You can just dooc cluster up --create-machine.

mikz commented 6 years ago

@fg78nc that is gone in 3.10.

bakayolo commented 6 years ago

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

$ oc version
oc v3.10.0+dd10d17
kubernetes v1.10.0+b81c8f8
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v3.10.0+e3465d0-44
kubernetes v1.10.0+b81c8f8

Still same issue.

$ oc cluster up --create-machine
Error: unknown flag: --create-machine

bakayolo commented 6 years ago

Confirmed working with

docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:25 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: true

rebeccaSimmonds19 commented 6 years ago

I still have this problem with docker q8.06.01-ce-mac73 and 3.10. Only updated as i thought this was fixed.

adambkaplan commented 6 years ago

FYI OpenShift 4.0 won't run "bare" on macOS, period. Minishift is the only viable path to running a local cluster on macOS at this point.

@csrwng once minishift supports 4.0 we can probably close this issue.

mikz commented 6 years ago

@adambkaplan minishift with hyperkit runner is basically the same as Docker for macOS. And that works. So, why it would not run on plain Docker for mac? Just because of some proxy setting?

leetrout commented 6 years ago

@adambkaplan Is your comment summing this up as a docker for mac issue and not an OpenShift issue at this point (due to the proxy settings being ignored by Docker for Mac)?

Could you elaborate by what you mean by "bare" on Mac OS? You are just reiterating some form of virtualization is required to provide a Linux kernel?

adambkaplan commented 6 years ago

@leetrout @mikz for OpenShift 3.11, the barrier is the outstanding docker for mac issue, which has been there for quite some time. Running "bare" via oc cluster up (which is what I meant) will fail if you're using Docker for Mac > 17.09.

OpenShift 4.0 (which is currently alpha.0) is a whole different ballgame. We don't have support for running local clusters on a Mac at present.

riuvshin commented 6 years ago

@adambkaplan what about Linux? would it still be possible to run oc cluster up ? that is a significant change that will break a lot of CI testing pipelines where products are tested on local openshift clusters. For testing purposes there is no way to run whole proper OS cluster and in the same time you can't run minishift with VM backend on CI infras due to nested virtualization issues.

adambkaplan commented 6 years ago

@riuvshin cluster up works on RHEL/CentOS/Fedora, and in principle any other Linux distro that supports docker and/or cri-o.

FYI cluster up is going away with 4.0. I'd refer you to openshift/installer if you want to follow how 4.0 installations will work.

riuvshin commented 6 years ago

@adambkaplan would it be possible to run local cluster as a set of docker containers with https://github.com/openshift/installer?

hakdogan commented 5 years ago

Hi.

I have same issue with docker 18.06.1-ce-mac73 (26764)

I try running a cluster with oc cluster up --loglevel=5 --http-proxy=gateway.docker.internal:3128 --https-proxy=gateway.docker.internal:3129 and I'm getting this warning.

WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list. 172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

The ~/Library/Group\ Containers/group.com.docker/http_proxy.json file has the following content:

{
  "exclude": "172.30.1.1",
  "transparent_http_ports": [
    80
  ],
  "transparent_https_ports": [
    443
  ]
}

Does this problem have a solution other than downgrade?

uLan08 commented 5 years ago

@hakdogan

Still using origin v3.6 but using docker-machine works for me

$ eval $(docker-machine env openshift) 
# needs pre created docker-machine, in my was case it was created by oc cluster up --create-machine

$ env | grep -i docker
# just to check that we are referencing the docker machine

$ oc cluster up 
# will create cluster in docker-machine

Note: I don't have a good way of persisting data. Stopping the docker-machine will wipe openshift cluster data

bobbydeveaux commented 5 years ago

If you head over to okd.io and download minishift, it uses the latest version of Openshift Origin...

$ minishift start $ minishift oc-env $ oc login etc

Hate having the VM but it works.

hakdogan commented 5 years ago

@uLan08 Hi Patrick

Unfortunately this didn't work for me. I got following error when executing the command.

oc cluster up --create-machine --loglevel=5

...
FAIL
   Error: could not create OpenShift configuration
   Caused By:
     Error: cannot start container f40c3852875bacceac537247005a516b1a9754ed0a5f10f88ed4600707f52a70
     Caused By:
       Error: Error response from daemon: {"message":"linux mounts: path /var/lib/origin/openshift.local.volumes is mounted on / but it is not a shared or slave mount"}

@bobbydeveaux

Hi Bobby,

Thank you for your advice but I don't want to use minishift.

Asgoret commented 5 years ago

Issue is still here((( When it will be fixed? Minishift haven't got actual documentation. All example explained via oc cluster up. I can't complete my research about ASB because oc doens't work and documentation for minishift doesn't work too or it's 404 on pages!

@mfojtik Hi! Do you have any additional information?

riuvshin commented 5 years ago

@Asgoret I used to downgrade docker for mac to get this worked...

Asgoret commented 5 years ago

@Asgoret I used to downgrade docker for mac to get this worked...

@riuvshin I upgrade Docker for macOS today....so which version i work?

Asgoret commented 5 years ago

@alberttwong one question. If oc cluster up won't work on macOS is that mean, that documentation will be corrected for minishift in short time? At this moment there is no actual documentation for large part of OKD in minishift. For example ASB installer documentation shows 404 error and minishift addons doesn't updates for 7 months or more time.

hakdogan commented 5 years ago

In addition to the thought of @Asgoret, I think that also the OKD documentation is not enough. There is no explanation for Mac users about this problem. I used to downgrade because there are no other options.

riuvshin commented 5 years ago

@Asgoret I was using this version https://download.docker.com/mac/stable/18433/Docker.dmg this is 17.06.0-ce-mac18

rb2010 commented 5 years ago

still an issue with docker for mac and oc cluster up: Docker version 18.09.1, build 4c52b90 will try to downgrade and check if it works.

benweizhu commented 5 years ago

Any updates?

benweizhu commented 5 years ago

I downgrade to Docker version 17.09.1-ce, build 19e2cf6, it works~

openshift-bot commented 5 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

infinitewarp commented 5 years ago

/remove-lifecycle stale

openshift-bot commented 4 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented 4 years ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

infinitewarp commented 4 years ago

/remove-lifecycle rotten

openshift-bot commented 4 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

infinitewarp commented 4 years ago

/remove-lifecycle stale

openshift-bot commented 4 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented 4 years ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten /remove-lifecycle stale

openshift-bot commented 4 years ago

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen. Mark the issue as fresh by commenting /remove-lifecycle rotten. Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci-robot commented 4 years ago

@openshift-bot: Closing this issue.

In response to [this](https://github.com/openshift/origin/issues/18596#issuecomment-678696174): >Rotten issues close after 30d of inactivity. > >Reopen the issue by commenting `/reopen`. >Mark the issue as fresh by commenting `/remove-lifecycle rotten`. >Exclude this issue from closing again by commenting `/lifecycle frozen`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

openshift / origin

cluster up fails to push to the registry on Docker for Mac CE 17.12.0 and newer #18596

Details

switch back to your machines context do: