search

projectatomic / adb-utils

A set of utilities for managing services used provided in the Atomic Developer Bundle.
GNU General Public License v2.0
13 stars 22 forks source link

Fix #149 Adding proxy support for docker and openshift #150

Closed praveenkumar closed 8 years ago

praveenkumar commented 8 years ago

This patch will put proxy configuration for docker daemon and add it to openshift environment variables.

hferentschik commented 8 years ago

How would the user set the env variables? AFAIU sccli configures OpenShift for proxy support, if the PROXY env variable is set. How do we do this?

Do we let the user add some provisioning to the Vagrantfile or do we let him add vagrant-proxy-conf or do we add something to vagrant-service-manager. vagrant-proxy-conf seems easy, but I feel a bit reluctant to pull in so many plugins. Not sure.

Also, I guess something similar is needed for Docker, right?

praveenkumar commented 8 years ago

How would the user set the env variables? AFAIU sccli configures OpenShift for proxy support, if the PROXY env variable is set. How do we do this?

sccli will only configure proxy support (openshift and docker both) using environment variable (PROXY, PROXY_USER, PROXY_PASSWORD) as per this PR.

Do we let the user add some provisioning to the Vagrantfile

Yes we have to add this ENV to our Vagrantfile as comments and when user need they just can comment out those ENV and able to use.

I feel a bit reluctant to pull in so many plugins.

Same here.

LalatenduMohanty commented 8 years ago

@praveenkumar We need to add documentation with this PR. The documentation should explain how user can just modify the openshift_option file to add support for proxy server.

coolbrg commented 8 years ago

It looks like documentation is not added yet.

praveenkumar commented 8 years ago

@budhrg I did a scratch build of ADB box and a custom Vagrantfile to test this out.

coolbrg commented 8 years ago

@praveenkumar Will test it out

coolbrg commented 8 years ago

@praveenkumar I did a testing with the provided box and Vagrantfile.

I just downloaded box and Vagrantfile and did vagrant up.

My observations:

[vagrant@centos7-adb ~]$ cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://fedora:fedora123@10.70.49.109:3128" "NO_PROXY=localhost,127.0.0.1,::1,.xip.io,172.30.85.77"

I hope this test out the proxy code.

If it is then it works :+1:

Let me know if any other steps need to be verified.

praveenkumar commented 8 years ago

Removed the HTTP_PROXY var from http-proxy.conf file followed by "daemon-reload" and "docker restart" Did docker pull busybox. Image got download (Removed previous image).

I think that should not work basically you have to enable iptable rule so that you can only allow traffic from proxy server => https://gist.github.com/LalatenduMohanty/5062050ac78721eda9676bbc15ecfd25

LalatenduMohanty commented 8 years ago

Some update on the testing. Docker pulls are working fine with with PR. But in my setup OpenShift is still not working as oc new-app cakephp-example is not triggering the build.

[vagrant@centos7-adb ~]$ oc new-app cakephp-example 
--> Deploying template "cakephp-example" in project "openshift" for "cakephp-example"
     With parameters:
      Memory Limit=512Mi
      Git Repository URL=https://github.com/openshift/cakephp-ex.git
      Git Reference=
      Context Directory=
      Application Hostname=
      GitHub Webhook Secret=bg4IeAq0EEQro6g5oYBEqThPxDwbkKTkjnkBU3OB # generated
      Database Service Name=
      Database Engine=
      Database Name=
      Database User=
      Database Password=
      CakePHP Secret Token=Rjj6eVtVe6eLwmVzWFq9m33Pm9wwG7IZzvQiSb_vkzaNMgN3vU # generated
      CakePHP Security Salt=Ot0osojrXi86niRhfG2QD6vurxqHHTMV1PFyJCSa # generated
      CakePHP Security Cipher Seed=138517866363011140632212858500 # generated
      OPcache Revalidation Frequency=2
--> Creating resources with label app=cakephp-example ...
    service "cakephp-example" created
    route "cakephp-example" created
    imagestream "cakephp-example" created
    buildconfig "cakephp-example" created
    deploymentconfig "cakephp-example" created
--> Success
    Build scheduled, use 'oc logs -f bc/cakephp-example' to track its progress.
    Run 'oc status' to view your app.
$ oc logs -f bc/cakephp-example
error: no builds found for "cakephp-example"

[vagrant@centos7-adb ~]$ oc status -v
In project OpenShift sample project (sample-project) on server https://10.1.2.2:8443

http://cakephp-example-sample-project.centos7-adb.10.1.2.2.xip.io (svc/cakephp-example)
  dc/cakephp-example deploys istag/cakephp-example:latest <-
    bc/cakephp-example builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6 
      not built yet
    deployment #1 waiting on image or update

http://nodejs-example-sample-project.centos7-adb.10.1.2.2.xip.io (svc/nodejs-example)
  dc/nodejs-example deploys istag/nodejs-example:latest <-
    bc/nodejs-example builds https://github.com/openshift/nodejs-ex.git with openshift/nodejs:0.10 
      not built yet
    deployment #1 waiting on image or update

Warnings:
  * istag/cakephp-example:latest needs to be imported or created by a build.
    try: oc start-build bc/cakephp-example
  * istag/nodejs-example:latest needs to be imported or created by a build.
    try: oc start-build bc/nodejs-example
  * The image trigger for dc/cakephp-example will have no effect until istag/cakephp-example:latest is imported or created by a build.
  * The image trigger for dc/nodejs-example will have no effect until istag/nodejs-example:latest is imported or created by a build.

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.
[vagrant@centos7-adb ~]$ oc get all
NAME              TYPE                                                         FROM          LATEST
cakephp-example   Source                                                       Git           0
nodejs-example    Source                                                       Git           0
NAME              DOCKER REPO                                                  TAGS          UPDATED
cakephp-example   172.30.252.223:5000/sample-project/cakephp-example                         
nodejs-example    172.30.252.223:5000/sample-project/nodejs-example                          
NAME              REVISION                                                     REPLICAS      TRIGGERED BY
cakephp-example   0                                                            1             config,image(cakephp-example:latest)
nodejs-example    0                                                            1             config,image(nodejs-example:latest)
NAME              HOST/PORT                                                    PATH          SERVICE           TERMINATION   LABELS
cakephp-example   cakephp-example-sample-project.centos7-adb.10.1.2.2.xip.io                 cakephp-example                 app=cakephp-example,template=cakephp-example
nodejs-example    nodejs-example-sample-project.centos7-adb.10.1.2.2.xip.io                  nodejs-example                  app=nodejs-example,template=nodejs-example
NAME              CLUSTER-IP                                                   EXTERNAL-IP   PORT(S)           AGE
cakephp-example   172.30.207.31                                                <none>        8080/TCP          28m
nodejs-example    172.30.119.116                                               <none>        8080/TCP          38m
LalatenduMohanty commented 8 years ago

In system logs I can see below logs

Jul 08 05:28:12 centos7-adb sh[15840]: E0708 09:28:12.530449   15923 config_controller.go:76] error instantiating Build from BuildConfig sample-project/cakephp-example: imagestreamtags "php:5.6" not found
Jul 08 05:28:12 centos7-adb sh[15840]: E0708 09:28:12.541356   15923 config_controller.go:76] error instantiating Build from BuildConfig sample-project/cakephp-example: imagestreamtags "php:5.6" not found
Jul 08 05:28:12 centos7-adb sh[15840]: E0708 09:28:12.554458   15923 config_controller.go:76] error instantiating Build from BuildConfig sample-project/cakephp-example: imagestreamtags "php:5.6" not found
Jul 08 05:28:12 centos7-adb sh[15840]: E0708 09:28:12.563414   15923 config_controller.go:76] error instantiating Build from BuildConfig sample-project/cakephp-example: imagestreamtags "php:5.6" not found
praveenkumar commented 8 years ago

@LalatenduMohanty have we got anything from mailing list about why you are facing this issue or what to be added in iptable rule?

LalatenduMohanty commented 8 years ago

As per https://docs.openshift.com/enterprise/3.0/admin_guide/http_proxies.html S2I and source template needs some change to properly work in a proxy setup. So I am trying to validate that.

praveenkumar commented 8 years ago

@LalatenduMohanty let me know if that changes works for you so we can make changes in the code also.

praveenkumar commented 8 years ago

Apart from above PR we also found some more setting to do to make it work with proxy.

1 Run openshift container with proxy environment 

start_openshift() {
        docker run $1 --name "openshift" --privileged --net=host --pid=host \
        -w ${ORIGIN_DIR} \
        -e "KUBECONFIG=${OPENSHIFT_DIR}/admin.kubeconfig" \
        -e "NO_PROXY=${NO_PROXY}" \
        -e "HTTP_PROXY=${HTTP_PROXY}" \
        -e "HTTPS_PROXY=${HTTP_PROXY}" \
        -v /:/rootfs:ro \
        -v /var/run:/var/run:rw \
        -v /sys:/sys:ro \
        -v /var/lib/docker:/var/lib/docker:rw \
        -v ${ORIGIN_DIR}/openshift.local.volumes:${ORIGIN_DIR}/openshift.local.volumes:z \
        -v ${ORIGIN_DIR}/openshift.local.config:${ORIGIN_DIR}/openshift.local.config:z \
        -v ${ORIGIN_DIR}/openshift.local.etcd:${ORIGIN_DIR}/openshift.local.etcd:z \
        $IMAGE start "${@:2}"
}

2 We should have hostname also part of NO_PROXY variable

NO_PROXY=127.0.0.1,10.0.2.15,10.1.2.2,172.17.0.1,.xip.io,172.30.0.0/16,172.17.0.0/16,centos7-adb

3 master-config.yaml file also should be updated with proxy variables.

kubernetesMasterConfig:
  admissionConfig:
    pluginConfig:
      BuildDefaults:
        configuration:
          apiVersion: v1
          env:
          - name: HTTP_PROXY
            value: http://fedora:fedora123@10.70.49.109:3128
          - name: HTTPS_PROXY
            value: http://fedora:fedora123@10.70.49.109:3128
          - name: NO_PROXY
            value: 127.0.0.1,10.0.2.15,10.1.2.2,172.17.0.1,.xip.io,172.30.0.0/16,172.1.0.0/16,centos7-adb
          gitHTTPProxy: http://fedora:fedora123@10.70.49.109:3128
          gitHTTPSProxy: http://fedora:fedora123@10.70.49.109:3128
          kind: BuildDefaultsConfig
  apiLevels: null
praveenkumar commented 8 years ago

New box will be available https://cbs.centos.org/koji/taskinfo?taskID=99676 Vagrant file: https://gist.github.com/praveenkumar/6aad634ba038044865bea9557631d74e

@LalatenduMohanty @budhrg please give it a try once build complete

coolbrg commented 8 years ago

@praveenkumar Some failures on pulling images:

Testing logs ``` [vagrant@centos7-adb ~]$ docker pull busybox Using default tag: latest Trying to pull repository docker.io/library/busybox ... latest: Pulling from docker.io/library/busybox 8ddc19f16526: Pull complete Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6 Status: Downloaded newer image for docker.io/busybox:latest [vagrant@centos7-adb ~]$ oc new-app cakephp-example --> Deploying template "cakephp-example" in project "openshift" for "cakephp-example" With parameters: Memory Limit=512Mi Git Repository URL=https://github.com/openshift/cakephp-ex.git Git Reference= Context Directory= Application Hostname= GitHub Webhook Secret=VsWTnlxqJr4IE1KLPVx6l8Fdyw284gXsbGhPK6Ln # generated Database Service Name= Database Engine= Database Name= Database User= Database Password= CakePHP Secret Token=VDtt7eEMzvM1cIQ_TvHthsX8tyrlnlPU7B_S3vWK5s32g04jty # generated CakePHP Security Salt=Uyk4TLUaD4MUfGcsgJ1lHh4cxsn7iAV1up7H28bK # generated CakePHP Security Cipher Seed=348461185423417014865221043661 # generated OPcache Revalidation Frequency=2 --> Creating resources with label app=cakephp-example ... service "cakephp-example" created route "cakephp-example" created imagestream "cakephp-example" created buildconfig "cakephp-example" created deploymentconfig "cakephp-example" created --> Success Build scheduled, use 'oc logs -f bc/cakephp-example' to track its progress. Run 'oc status' to view your app. [vagrant@centos7-adb ~]$ oc logs -f bc/cakephp-example I0718 08:50:16.335944 1 docker.go:351] Image "registry.access.redhat.com/rhscl/php-56-rhel7:latest" not available locally, pulling ... I0718 09:03:25.879755 1 source.go:197] Downloading "https://github.com/openshift/cakephp-ex.git" ... W0718 09:04:07.752261 1 common.go:89] An error occurred saving build revision: Put https://172.30.0.1:443/oapi/v1/namespaces/sample-project/builds/cakephp-example-1/details: dial tcp 172.30.0.1:443: i/o timeout I0718 09:04:07.752363 1 install.go:251] Using "assemble" installed from "/.s2i/bin/assemble" I0718 09:04:07.752416 1 install.go:251] Using "run" installed from "image:///usr/libexec/s2i/run" I0718 09:04:07.752436 1 install.go:251] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts" ---> Installing application source... I0718 09:04:28.875370 1 sti.go:267] Using provided push secret for pushing 172.30.74.67:5000/sample-project/cakephp-example:latest image I0718 09:04:28.875421 1 sti.go:271] Pushing 172.30.74.67:5000/sample-project/cakephp-example:latest image ... E0718 09:04:31.895787 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... E0718 09:04:39.906771 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... E0718 09:04:47.915609 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... E0718 09:04:55.933410 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... E0718 09:05:03.949483 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... E0718 09:05:12.001447 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... E0718 09:05:20.020365 1 dockerutil.go:86] push for image 172.30.74.67:5000/sample-project/cakephp-example:latest failed, will retry in 5s ... I0718 09:05:25.020775 1 sti.go:276] Registry server Address: I0718 09:05:25.020806 1 sti.go:277] Registry server User Name: serviceaccount I0718 09:05:25.020814 1 sti.go:278] Registry server Email: serviceaccount@example.org I0718 09:05:25.020820 1 sti.go:283] Registry server Password: <> F0718 09:05:25.020833 1 builder.go:204] Error: build error: Failed to push image. Response from registry is: unable to ping registry endpoint https://172.30.74.67:5000/v0/ v2 ping attempt failed with error: Get https://172.30.74.67:5000/v2/: dial tcp 172.30.74.67:5000: connection refused v1 ping attempt failed with error: Get https://172.30.74.67:5000/v1/_ping: dial tcp 172.30.74.67:5000: connection refused ```
coolbrg commented 8 years ago

I followed the commands to setup my host for proxy through LalatenduMohanty gist. Note: Through these steps, you will lost with internet through LAN or Wifi. Setup proxy through firefox.

Then starting with the box and Vagrantfile from praveen's comment.

I performed:

$ vagrant add <test-build-box>
$ vagrant up             
$ vagrant ssh

Observations

Logs ``` [vagrant@centos7-adb ~]$ oc new-app cakephp-example --> Deploying template "cakephp-example" in project "openshift" for "cakephp-example" With parameters: Memory Limit=512Mi Git Repository URL=https://github.com/openshift/cakephp-ex.git Git Reference= Context Directory= Application Hostname= GitHub Webhook Secret=wvKY88OcCIqpBsTakqcOh7xNxF612G6yQPbhm2Bw # generated Database Service Name= Database Engine= Database Name= Database User= Database Password= CakePHP Secret Token=zoYEubFo0NvDjVyzm9eTbDbjkLBXf0ndE828Zer6YS8Qt9yL9D # generated CakePHP Security Salt=WIRqdgJHHtOit5Kk3mDwU7mfguCwS0lMOsWp6rN0 # generated CakePHP Security Cipher Seed=448571761832472586324008500258 # generated OPcache Revalidation Frequency=2 --> Creating resources with label app=cakephp-example ... service "cakephp-example" created route "cakephp-example" created imagestream "cakephp-example" created buildconfig "cakephp-example" created deploymentconfig "cakephp-example" created --> Success Build scheduled, use 'oc logs -f bc/cakephp-example' to track its progress. Run 'oc status' to view your app. [vagrant@centos7-adb ~]$ oc logs -f bc/cakephp-example I0718 11:27:39.313075 1 docker.go:351] Image "registry.access.redhat.com/rhscl/php-56-rhel7:latest" not available locally, pulling ... I0718 11:33:48.540295 1 source.go:197] Downloading "https://github.com/openshift/cakephp-ex.git" ... I0718 11:33:59.932152 1 install.go:251] Using "assemble" installed from "/.s2i/bin/assemble" I0718 11:33:59.932500 1 install.go:251] Using "run" installed from "image:///usr/libexec/s2i/run" I0718 11:33:59.932533 1 install.go:251] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts" ---> Installing application source... I0718 11:34:17.853607 1 sti.go:267] Using provided push secret for pushing 172.30.111.237:5000/sample-project/cakephp-example:latest image I0718 11:34:17.853654 1 sti.go:271] Pushing 172.30.111.237:5000/sample-project/cakephp-example:latest image ... I0718 11:35:47.513179 1 sti.go:287] Successfully pushed 172.30.111.237:5000/sample-project/cakephp-example:latest [vagrant@centos7-adb ~]$ oc status In project OpenShift sample project (sample-project) on server https://127.0.0.1:8443 http://cakephp-example-sample-project.centos7-adb.10.1.2.2.xip.io (svc/cakephp-example) dc/cakephp-example deploys istag/cakephp-example:latest <- bc/cakephp-example builds https://github.com/openshift/cakephp-ex.git with openshift/php:5.6 deployment #1 deployed 36 seconds ago - 1 pod View details with 'oc describe /' or list everything with 'oc get all'. [vagrant@centos7-adb ~]$ curl http://cakephp-example-sample-project.centos7-adb.10.1.2.2.xip.io # Printed the html page ```
praveenkumar commented 8 years ago

@budhrg Thanks for testing this out looks like basic testing working as expected.

LalatenduMohanty commented 8 years ago

I can deploy application with the new image @praveenkumar has given. However I am having issues accessing the application so looking in to it.

coolbrg commented 8 years ago

@praveenkumar @LalatenduMohanty I forgot to mention that I can access the application from inside VM(can able to curl) but from host it was not accessible(some how missed in hustle bustle of proxy setup)

LalatenduMohanty commented 8 years ago

@budhrg ok, what is the curl command you have used?

coolbrg commented 8 years ago

@LalatenduMohanty curl http://cakephp-example-sample-project.centos7-adb.10.1.2.2.xip.io full details in the comment observations section (click on the arrow)

LalatenduMohanty commented 8 years ago

It seems the issue is with Libvirt setup. This PR works fine with Virtualbox provider. The issue mostly is in the iptable rules we are using for simulating the proxy setup. We have spent enough time and still can not make application endpoint accessible in proxy setup + libvirt accessible. However the application deployed fine which can be seen in oc status command.

LalatenduMohanty commented 8 years ago

I am going to merge the PR as we want to get feedback from users about if this works in their proxy setup. Also we are kind of confident it should work in libvirt setup where the xip.io domain is working for the guest Vagrant box.

I will create a separate issue to track the libvirt issue.

LalatenduMohanty commented 8 years ago

https://github.com/projectatomic/adb-utils/issues/159 to track the libvirt proxy issue.