Introduce proxy_env variable

[mvdbeek]

The default proxy_env variable is empty, but users may modify their inventory file to include global environment variables, such as http_proxy and https_proxy.

A proxy_env variable may look like this:

proxy_env:
  http_proxy: http://proxy.bos.example.com:8080
  https_proxy: http://proxy.bos.example.com:8080

More documentation can be found in the Ansible Documentation at http://docs.ansible.com/ansible/playbooks_environment.html

[mvdbeek]

You can open a socks proxy on localhost, then point ansible to that socks proxy: ssh -D 8888 localhost open a proxy on port 8888. To confirm that is working, do

http_proxy=socks5://127.0.0.1:8888 https_proxy=socks5://127.0.0.1:8888 curl https://usegalaxy.org/api/version

which will succeed. Close the proxy (by closing the ssh session) and it will fail. Re-open the proxy, and now use

proxy_env:
   http_proxy: socks5://127.0.0.1:8888
   https_proxy: socks5://127.0.0.1:8888

and run a task like this:

command: curl https://usegalaxy.org/api/version
environment: '{{ proxy_env }}'

When the proxy is open, it should work, when its closed it should fail.

[drosofff]

OK with the task like this... but in the same context the GKS playbook fails because

TASK [galaxyprojectdotorg.galaxy-os : Add custom Galaxy PPA (used for nginx package)] ***
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Unsupported proxy scheme: socks5. Currently ansible only supports HTTP proxies."}

Thus there is no, in real, test

Maybe @colindaven can test the PR ?

[mvdbeek]

Thus there is no, in real, test

You can also specify some http proxy that doesn't exist. If the task fails the environment variable has been properly propagated.

[drosofff]

@mvdbeek

If the task fails the environment variable has been properly propagated.

This is not exactly what is called a test... with this approach I would have more success in mines :-)

I'll see whether I can manage to get a real proxy somewhere for a test

[drosofff]

I'll give a shot to this https://gist.github.com/webinista/812c20247a6c21e639ce

[mvdbeek]

So here's my test:

mvandenb@u934-bdd-4-89-6580:~/src/GalaxyKickStart (doc_proxy)*$ cat hosts
[proxy]
localhost ansible_connection=local

mvandenb@u934-bdd-4-89-6580:~/src/GalaxyKickStart (doc_proxy)*$ cat proxy.yml
- hosts: all

  environment: "{{ proxy_env }}"

  tasks:
      - command: 'curl https://usegalaxy.org/api/version'

mvandenb@u934-bdd-4-89-6580:~/src/GalaxyKickStart (doc_proxy)*$ cat group_vars/proxy
## Uses default settings defined in group_vars/all, but sets the `proxy_env` variable.

proxy_env:
  http_proxy: http://www-cache:3128
  https_proxy: http://www-cache:3128

Now with our local proxy in place:

mvandenb@u934-bdd-4-89-6580:~/src/GalaxyKickStart (doc_proxy)*$ ansible-playbook -i hosts proxy.yml

PLAY [all] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [command] ***********************************************************************************************************************************************************************
 [WARNING]: Consider using get_url or uri module rather than running curl

changed: [localhost]

PLAY RECAP ***************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Remove the proxy connection and the playbook fails:

mvandenb@u934-bdd-4-89-6580:~/src/GalaxyKickStart (doc_proxy)*$ ansible-playbook -i hosts proxy.yml

PLAY [all] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [command] ***********************************************************************************************************************************************************************
 [WARNING]: Consider using get_url or uri module rather than running curl

fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["curl", "https://usegalaxy.org/api/version"], "delta": "0:00:00.024863", "end": "2017-05-12 15:30:52.329555", "failed": true, "rc": 5, "start": "2017-05-12 15:30:52.304692", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (5) Could not resolve proxy: www-cache", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (5) Could not resolve proxy: www-cache"], "stdout": "", "stdout_lines": []}
    to retry, use: --limit @/Users/mvandenb/src/GalaxyKickStart/proxy.retry

PLAY RECAP ***************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1

[drosofff]

I have a squid proxy in the cloud and an ansible target vm running behind the proxy vm. @mvdbeek although your test works perfectly under these settings there is still errors to update the apt cache in with galaxy-os role. I may merge this PR, but we need a real test to the playbook behind a proxy

[drosofff]

So here is the progress with a squid proxy open on port 3128 group_vars/proxy:

proxy_env:
  http_proxy: http://instance-1:3128
  https_proxy: http://instance-1:3128

and inventory_files/galaxy-kickstart

[kickstart]
localhost ansible_connection=local
[proxy]
localhost ansible_connection=local

Then, Adding

# Install system packages
- name: Add Ubuntu repository key
  apt_key: keyserver=hkp://keyserver.ubuntu.com:80 id=E9708E5E0DEE60262A3FE2089EB12D87B9BDC5ED

in roles/galaxyprojectdotorg.galaxy-os/tasks/ubuntu/packages.yml

allows now to go up to the galaxy-tools role, which fails at:

TASK [galaxyprojectdotorg.galaxy-tools : Create Galaxy bootstrap user] *********
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/home/galaxy/galaxy/.venv/bin/python", "manage_bootstrap_user.py", "-c", "/home/galaxy/galaxy/config/galaxy.ini", "create", "-e", "admin@galaxy.org", "-u", "cloud", "-p", "admin", "-a", "admin"], "delta": "0:00:00.209513", "end": "2017-05-13 20:39:47.989883", "failed": true, "rc": 1, "start": "2017-05-13 20:39:47.780370", "stderr": "Traceback (most recent call last):\n  File \"manage_bootstrap_user.py\", line 19, in <module>\n    import galaxy.config\n  File \"/home/galaxy/galaxy/lib/galaxy/config.py\", line 26, in <module>\n    from galaxy.util.postfork import register_postfork_function\n  File \"/home/galaxy/galaxy/lib/galaxy/util/postfork.py\", line 10, in <module>\n    if uwsgi.numproc:\nAttributeError: 'module' object has no attribute 'numproc'", "stdout": "", "stdout_lines": [], "warnings": []}

Its ok without proxy but after trying with proxy, reverting to no-proxy does not revert this error...

[drosofff]

OK, I figured out the problem.

@mvdbeek one have to had in group_vars/proxy:

## Uses default settings defined in group_vars/all, but sets the `proxy_env` variable.

proxy_env:
  http_proxy: http://proxy.bos.example.com:8080
  https_proxy: http://proxy.bos.example.com:8080
  no_proxy: localhost,127.0.0.0,127.0.1.1,127.0.1.1,local.home

[mvdbeek]

Ah, that makes sense. I'll add that

On May 14, 2017 12:42 PM, "Christophe Antoniewski" notifications@github.com wrote:

OK, I figured out the problem.

@mvdbeek https://github.com/mvdbeek one have to had in group_vars/proxy:

Uses default settings defined in group_vars/all, but sets the proxy_env variable.

proxy_env: http_proxy: http://proxy.bos.example.com:8080 https_proxy: http://proxy.bos.example.com:8080 no_proxy: localhost,127.0.0.0,127.0.1.1,127.0.1.1,local.home

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ARTbio/GalaxyKickStart/pull/236#issuecomment-301304037, or mute the thread https://github.com/notifications/unsubscribe-auth/AGfVpX3Dr54MDaW9gfikXNMlTm-4E87Zks5r5tqMgaJpZM4NYMMM .

[drosofff]

👍 For records, the squid docker container used to test with proxy was https://github.com/sameersbn/docker-squid