git clone causes `npm i aframe` to hang in container on PWD

[stealthybox]

Use case

I have a Dockerfile that I was using for some WebVR development. I was going to demo it on my phone by using Play-With-Docker.

Initial Symptoms

• made a new session on play-with-docker.com
• pulled stealthybox/vr-swarm
• swarm init
• attempted to docker build my image: --> npm install hangs while installing aframe dependencies from npm and github.

@xetorthio, @drmdrew, and I chatted about this in the community slack.

Reproducing

I have reduced the scope to just installing aframe. The following command:

docker run --rm -itd --entrypoint sh node:alpine -c 'apk add --update git && npm i -ddd aframe'
docker logs -f `docker ps -q | head -n1`

runs properly on:

• docker-for-windows(17.05.0-ce)
• google-cloud-shell (1.11.2)

It fails to run on:

• play-with-docker(17.05.0-ce) # hangs afterAdd tween.js

looking at ps, there are active two active processes for git clone related to npm:

ps
PID   USER     TIME   COMMAND
    1 root       0:00 /bin/sh -c cat /etc/hosts >/etc/hosts.bak &&     sed 's/^::1.*//' /etc/hosts.bak > /etc/hosts &&     sed -i "s/\DOCKER_STORAGE_DRIVER/$DOCKER_STORAGE_DRIVER/" /etc/docker/daemon.json &&     sed -i "s/\PWD_IP_ADDRESS/$PWD_IP_ADDRESS/" /etc/docker/daemon.json &&     sed -i "s/\DOCKER_TLSE
    7 root       0:05 dockerd
    8 root       0:00 script -q -c /bin/bash -l /dev/null
   10 root       0:00 sh -c /bin/bash -l
   12 root       0:00 /bin/bash -l
   36 root       0:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --debug
  343 root       0:00 docker-containerd-shim d99cb1fd481f623b3fc289cb1dbf3b0aeecd82b3a8d6cd4e0002f999a07e883b /var/run/docker/libcontainerd/d99cb1fd481f623b3fc289cb1dbf3b0aeecd82b3a8d6cd4e0002f999a07e883b docker-runc
  360 root       0:00 sh -c apk add --update git && npm i -ddd aframe
  420 root       0:02 npm
  430 root       0:00 git clone --template=/root/.npm/_git-remotes/_templates --mirror git://github.com/dmarcos/document-register-element.git /root/.npm/_git-remotes/git-github-com-dmarcos-document-register-element-git-8ccc532b7-4e079fcb
  431 root       0:00 git clone --template=/root/.npm/_git-remotes/_templates --mirror git://github.com/dmarcos/webvr-polyfill.git /root/.npm/_git-remotes/git-github-com-dmarcos-webvr-polyfill-git-a02a8089b-646b18fa
 1735 root       0:00 ps

Cloning Git Repos on PWD

I made a discovery. You can clone the repos fine from the host using the git protocol:

[node1] (local) root@10.0.26.3 ~
$ git clone git://github.com/dmarcos/document-register-element.git && git clone git://github.com/dmarcos/webvr-polyfill.git
Cloning into 'document-register-element'...
remote: Counting objects: 1028, done.
remote: Total 1028 (delta 0), reused 0 (delta 0), pack-reused 1028
Receiving objects: 100% (1028/1028), 1.62 MiB | 0 bytes/s, done.
Resolving deltas: 100% (614/614), done.
Cloning into 'webvr-polyfill'...
remote: Counting objects: 4061, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4061 (delta 1), reused 1 (delta 1), pack-reused 4058
Receiving objects: 100% (4061/4061), 12.20 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2149/2149), done.

However, you cannot from inside an alpine or alpine:node container:

docker run -it alpine ash -c 'apk add --update git && git clone git://github.com/stealthybox/docker-demo'
# hangs

... but it works using https:

docker run -it alpine ash -c 'apk add --update git && git clone https://github.com/stealthybox/docker-demo'
# succeeds

ubuntu hangs after this strange message:

docker run -it ubuntu bash -c 'apt-get update && apt-get install -y git && git clone git://github.com/stealthybox/docker-demo'
...

Cloning into 'docker-demo'...
warning: expected SRV RR, found RR type 1
warning: expected SRV RR, found RR type 1
warning: expected SRV RR, found RR type 1
warning: expected SRV RR, found RR type 1

121 suggests that this is not intentional.

Perhaps there some sort of container specific IPTables / firewalls preventing this? ( Maybe allowing only HTTP/S? ) If so we should document and/or fix it because it hangs silently.

As always, thanks for pushing to make pwd better for the community.

Cheers, Leigh 🐳

[jpetazzo]

We definitely had this problem a while ago. I don't know what caused it, because somebody (@xetorthio or @marcosnils) did something that fixed it magically; and I don't know what something was.

Ideas to investigate:

• run tcpdump on the host to see what kind of packets go around when trying a git clone git://...
• check the output of iptables -nxvL before and after trying the git clone (and look which counters are incremented; it might give a hint about some iptables rule filtering the traffic)
• ditto with iptables -nxvL -t nat

[marcosnils]

hey @jpetazzo the magic thing you're talking about was when trying to close repos with the git:// url which was just adding a .gitconfig file in the PWD instance (https://github.com/play-with-docker/play-with-docker/blob/master/.gitconfig).

I have no clue why this specific package hangs the complete installation. Don't have much time in my hands lately, but I'll try to investigate whenever I can.

[xetorthio]

Yes. Just tried and a workaround would be to add to your ~/.gitconfig the following:

[url "https://"]   
    insteadOf = git://

[marcosnils]

@xetorthio but we're already shipping that .gitconfig file in PWD dind containers...

[stealthybox]

@marcosnils you'd need to mount the .gitconfig into any containers you run in the dind if you want to pull git:// repos from inside of them, though

[marcosnils]

@marcosnils you'd need to mount the .gitconfig into any containers you run in the dind if you want to pull git:// repos from inside of them, though

doh!. Yeah, I remember adding that .gitconfig in PWD for some other connection issues. Don't really understand why standard git:// doesn't work though.