search

jelaiw / hapi-lab

Virtual lab for Corey Ball's "Hacking APIs" book.
0 stars 1 forks source link

Fix juice shop Ansible playbook intermittent docker pull failures #32

Open jelaiw opened 1 year ago

jelaiw commented 1 year ago

Notes

jelaiw commented 1 year ago

Log verbose output.

$ ansible-playbook -i hosts -vvv juice-shop.yml 
ansible-playbook [core 2.15.3]
  config file = None
  configured module search path = ['/home/vscode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/py-utils/venvs/ansible-core/lib/python3.9/site-packages/ansible
  ansible collection location = /home/vscode/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/py-utils/bin/ansible-playbook
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (/usr/local/py-utils/venvs/ansible-core/bin/python)
  jinja version = 3.1.2
  libyaml = True
No config file found; using defaults
host_list declined parsing /workspaces/hapi-lab/ansible/hosts as it did not pass its verify_file() method
script declined parsing /workspaces/hapi-lab/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /workspaces/hapi-lab/ansible/hosts as it did not pass its verify_file() method
Parsed /workspaces/hapi-lab/ansible/hosts inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: juice-shop.yml *****************************************************************************************************************************************
1 plays in juice-shop.yml

PLAY [OWASP Juice Shop] ******************************************************************************************************************************************

--- SNIP ---

TASK [Pull image] ************************************************************************************************************************************************
task path: /workspaces/hapi-lab/ansible/juice-shop.yml:4
<34.71.170.170> ESTABLISH SSH CONNECTION FOR USER: vulnweb
<34.71.170.170> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vulnweb"' -o ConnectTimeout=10 -o 'ControlPath="/home/vscode/.ansible/cp/e4e736e240"' 34.71.170.170 '/bin/sh -c '"'"'echo ~vulnweb && sleep 0'"'"''
<34.71.170.170> (0, b'/home/vulnweb\n', b'')
<34.71.170.170> ESTABLISH SSH CONNECTION FOR USER: vulnweb
<34.71.170.170> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vulnweb"' -o ConnectTimeout=10 -o 'ControlPath="/home/vscode/.ansible/cp/e4e736e240"' 34.71.170.170 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/vulnweb/.ansible/tmp `"&& mkdir "` echo /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788 `" && echo ansible-tmp-1692641012.0381455-6137-62360178002788="` echo /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788 `" ) && sleep 0'"'"''
<34.71.170.170> (0, b'ansible-tmp-1692641012.0381455-6137-62360178002788=/home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788\n', b'')
Using module file /usr/local/py-utils/venvs/ansible-core/lib/python3.9/site-packages/ansible_collections/community/docker/plugins/modules/docker_image.py
<34.71.170.170> PUT /home/vscode/.ansible/tmp/ansible-local-6091h1fueuqm/tmpz_v1dgzk TO /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788/AnsiballZ_docker_image.py
<34.71.170.170> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vulnweb"' -o ConnectTimeout=10 -o 'ControlPath="/home/vscode/.ansible/cp/e4e736e240"' '[34.71.170.170]'
<34.71.170.170> (0, b'sftp> put /home/vscode/.ansible/tmp/ansible-local-6091h1fueuqm/tmpz_v1dgzk /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788/AnsiballZ_docker_image.py\n', b'')
<34.71.170.170> ESTABLISH SSH CONNECTION FOR USER: vulnweb
<34.71.170.170> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vulnweb"' -o ConnectTimeout=10 -o 'ControlPath="/home/vscode/.ansible/cp/e4e736e240"' 34.71.170.170 '/bin/sh -c '"'"'chmod u+x /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788/ /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788/AnsiballZ_docker_image.py && sleep 0'"'"''
<34.71.170.170> (0, b'', b'')
<34.71.170.170> ESTABLISH SSH CONNECTION FOR USER: vulnweb
<34.71.170.170> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vulnweb"' -o ConnectTimeout=10 -o 'ControlPath="/home/vscode/.ansible/cp/e4e736e240"' -tt 34.71.170.170 '/bin/sh -c '"'"'/usr/bin/python3 /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788/AnsiballZ_docker_image.py && sleep 0'"'"''
<34.71.170.170> (1, b'\r\n{"failed": true, "msg": "Error connecting: Error while fetching server API version: (\'Connection aborted.\', PermissionError(13, \'Permission denied\'))", "exception": "  File \\"/tmp/ansible_community.docker.docker_image_payload_za1x4okf/ansible_community.docker.docker_image_payload.zip/ansible_collections/community/docker/plugins/module_utils/common_api.py\\", line 117, in __init__\\n    super(AnsibleDockerClientBase, self).__init__(**self._connect_params)\\n  File \\"/tmp/ansible_community.docker.docker_image_payload_za1x4okf/ansible_community.docker.docker_image_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py\\", line 188, in __init__\\n    self._version = self._retrieve_server_version()\\n  File \\"/tmp/ansible_community.docker.docker_image_payload_za1x4okf/ansible_community.docker.docker_image_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py\\", line 212, in _retrieve_server_version\\n    raise DockerException(\\n", "invocation": {"module_args": {"name": "bkimminich/juice-shop", "source": "pull", "docker_host": "unix://var/run/docker.sock", "api_version": "auto", "timeout": 60, "tls": false, "use_ssh_client": false, "validate_certs": false, "debug": false, "force_source": false, "force_absent": false, "force_tag": false, "push": false, "state": "present", "tag": "latest", "tls_hostname": null, "ca_cert": null, "client_cert": null, "client_key": null, "ssl_version": null, "build": null, "archive_path": null, "load_path": null, "pull": null, "repository": null}}}\r\n', b'Shared connection to 34.71.170.170 closed.\r\n')
<34.71.170.170> Failed to connect to the host via ssh: Shared connection to 34.71.170.170 closed.
<34.71.170.170> ESTABLISH SSH CONNECTION FOR USER: vulnweb
<34.71.170.170> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="vulnweb"' -o ConnectTimeout=10 -o 'ControlPath="/home/vscode/.ansible/cp/e4e736e240"' 34.71.170.170 '/bin/sh -c '"'"'rm -f -r /home/vulnweb/.ansible/tmp/ansible-tmp-1692641012.0381455-6137-62360178002788/ > /dev/null 2>&1 && sleep 0'"'"''
<34.71.170.170> (0, b'', b'')
The full traceback is:
  File "/tmp/ansible_community.docker.docker_image_payload_za1x4okf/ansible_community.docker.docker_image_payload.zip/ansible_collections/community/docker/plugins/module_utils/common_api.py", line 117, in __init__
    super(AnsibleDockerClientBase, self).__init__(**self._connect_params)
  File "/tmp/ansible_community.docker.docker_image_payload_za1x4okf/ansible_community.docker.docker_image_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py", line 188, in __init__
    self._version = self._retrieve_server_version()
  File "/tmp/ansible_community.docker.docker_image_payload_za1x4okf/ansible_community.docker.docker_image_payload.zip/ansible_collections/community/docker/plugins/module_utils/_api/api/client.py", line 212, in _retrieve_server_version
    raise DockerException(
fatal: [vulnweb]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "archive_path": null,
            "build": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "docker_host": "unix://var/run/docker.sock",
            "force_absent": false,
            "force_source": false,
            "force_tag": false,
            "load_path": null,
            "name": "bkimminich/juice-shop",
            "pull": null,
            "push": false,
            "repository": null,
            "source": "pull",
            "ssl_version": null,
            "state": "present",
            "tag": "latest",
            "timeout": 60,
            "tls": false,
            "tls_hostname": null,
            "use_ssh_client": false,
            "validate_certs": false
        }
    },
    "msg": "Error connecting: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))"
}

PLAY RECAP *******************************************************************************************************************************************************
vulnweb                    : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
jelaiw commented 1 year ago 
$ sudo tail -f /var/log/messages 
Sep  4 16:34:43 debian kernel: [  543.804794] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Sep  4 16:34:43 debian kernel: [  543.821676] Bridge firewalling registered
Sep  4 16:34:43 debian kernel: [  544.007248] Initializing XFRM netlink socket
Sep  4 16:35:03 debian ansible-ansible.builtin.user: Invoked with name=vulnweb groups=['docker'] append=True state=present non_unique=False force=False remove=False create_home=True system=False move_home=False ssh_key_bits=0 ssh_key_type=rsa ssh_key_comment=ansible-generated on vulnweb update_password=always uid=None group=None comment=None home=None shell=None password=NOT_LOGGING_PARAMETER login_class=None password_expire_max=None password_expire_min=None hidden=None seuser=None skeleton=None generate_ssh_key=None ssh_key_file=None ssh_key_passphrase=NOT_LOGGING_PARAMETER expires=None password_lock=None local=None profile=None authorization=None role=None umask=None
Sep  4 16:35:49 debian ansible-ansible.legacy.setup: Invoked with gather_subset=['all'] gather_timeout=10 filter=[] fact_path=/etc/ansible/facts.d
Sep  4 16:35:51 debian ansible-community.docker.docker_image: Invoked with name=bkimminich/juice-shop source=pull docker_host=unix://var/run/docker.sock api_version=auto timeout=60 tls=False use_ssh_client=False validate_certs=False debug=False force_source=False force_absent=False force_tag=False push=False state=present tag=latest tls_hostname=None ca_cert=None client_cert=None client_key=None ssl_version=None build=None archive_path=None load_path=None pull=None repository=None
Sep  4 16:36:48 debian ansible-ansible.legacy.setup: Invoked with gather_subset=['all'] gather_timeout=10 filter=[] fact_path=/etc/ansible/facts.d
Sep  4 16:36:50 debian ansible-community.docker.docker_image: Invoked with name=bkimminich/juice-shop source=pull docker_host=unix://var/run/docker.sock api_version=auto timeout=60 tls=False use_ssh_client=False validate_certs=False debug=False force_source=False force_absent=False force_tag=False push=False state=present tag=latest tls_hostname=None ca_cert=None client_cert=None client_key=None ssl_version=None build=None archive_path=None load_path=None pull=None repository=None
Sep  4 16:38:05 debian ansible-ansible.legacy.setup: Invoked with gather_subset=['all'] gather_timeout=10 filter=[] fact_path=/etc/ansible/facts.d
Sep  4 16:38:07 debian ansible-community.docker.docker_image: Invoked with name=bkimminich/juice-shop source=pull docker_host=unix://var/run/docker.sock api_version=auto timeout=60 tls=False use_ssh_client=False validate_certs=False debug=False force_source=False force_absent=False force_tag=False push=False state=present tag=latest tls_hostname=None ca_cert=None client_cert=None client_key=None ssl_version=None build=None archive_path=None load_path=None pull=None repository=None
Sep  4 16:38:32 debian ansible-community.docker.docker_container: Invoked with name=juice-shop image=bkimminich/juice-shop ports=['80:3000'] published_ports=['80:3000'] docker_host=unix://var/run/docker.sock api_version=auto timeout=60 tls=False use_ssh_client=False validate_certs=False debug=False cleanup=False container_default_behavior=no_defaults command_handling=correct force_kill=False ignore_image=False image_comparison=desired-image image_label_mismatch=ignore image_name_mismatch=ignore keep_volumes=True networks_cli_compatible=True output_logs=False pull=False purge_networks=False recreate=False restart=False state=started tls_hostname=None ca_cert=None client_cert=None client_key=None ssl_version=None comparisons=None default_host_ip=None kill_signal=None paused=None removal_wait_timeout=None auto_remove=None blkio_weight=None capabilities=None cap_drop=None cgroupns_mode=None cgroup_parent=None command=None cpu_period=None cpu_quota=None cpuset_cpus=None cpuset_mems=None cpu_shares=None entrypoint=None cpus=None detach=None interactive=None devices=None device_read_bps=None device_write_bps=None device_read_iops=None device_write_iops=None device_requests=None dns_servers=None dns_opts=None dns_search_domains=None domainname=None env=None env_file=None etc_hosts=None groups=None healthcheck=None hostname=None init=None ipc_mode=None kernel_memory=None labels=None links=None log_driver=None log_options=None mac_address=None memory=None memory_reservation=None memory_swap=None memory_swappiness=None stop_timeout=None network_mode=None networks=None oom_killer=None oom_score_adj=None pid_mode=None pids_limit=None platform=None privileged=None read_only=None restart_policy=None restart_retries=None runtime=None security_opts=None shm_size=None stop_signal=None storage_opts=None sysctls=None tmpfs=None tty=None ulimits=None user=None userns_mode=None uts=None volume_driver=None volumes_from=None working_dir=None mounts=None volumes=None exposed_ports=None publish_all_ports=None
Sep  4 16:38:34 debian kernel: [  774.904663] docker0: port 1(vethcad201b) entered blocking state
Sep  4 16:38:34 debian kernel: [  774.910775] docker0: port 1(vethcad201b) entered disabled state
Sep  4 16:38:34 debian kernel: [  774.917251] device vethcad201b entered promiscuous mode
Sep  4 16:38:34 debian kernel: [  774.923072] docker0: port 1(vethcad201b) entered blocking state
Sep  4 16:38:34 debian kernel: [  774.930504] docker0: port 1(vethcad201b) entered forwarding state
Sep  4 16:38:34 debian kernel: [  774.940027] docker0: port 1(vethcad201b) entered disabled state
Sep  4 16:38:35 debian kernel: [  775.270716] eth0: renamed from vethfba39d5
Sep  4 16:38:35 debian kernel: [  775.290475] IPv6: ADDRCONF(NETDEV_CHANGE): vethcad201b: link becomes ready
Sep  4 16:38:35 debian kernel: [  775.297770] docker0: port 1(vethcad201b) entered blocking state
Sep  4 16:38:35 debian kernel: [  775.304470] docker0: port 1(vethcad201b) entered forwarding state
Sep  4 16:38:35 debian kernel: [  775.311403] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready

Unclear why the docker pull failed twice at 16:35:51 and 16:36:50, but succeeded at 16:38:07. Note that /var/run/docker.sock and /var/run/docker.pid files exist.