search

moby / moby

The Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
https://mobyproject.org/
Apache License 2.0
68.49k stars 18.63k forks source link

docker build fails using Git Enterprise URL #10810

Closed marcellodesales closed 7 years ago

marcellodesales commented 9 years ago

Public GitHub builds work

Given that a Git repository has a Dockerfile in its root path, I learned that we can build a Docker image from a GitHub URL as follows:

 sudo docker build github.com/mikaelhg/docker-rabbitmq.git
Sending build context to Docker daemon 176.1 kB
Sending build context to Docker daemon 
Step 0 : FROM ubuntu:14.04
 ---> ed5a78b7b42b
Step 1 : MAINTAINER Mikael Gueck "gumi@iki.fi"
 ---> Using cache
 ---> 5b52159a47de
Step 2 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 8d3e60172f2a
Step 3 : ADD rabbitmq-signing-key-public.asc /tmp/rabbitmq-signing-key-public.asc
 ---> 4947194a6f2f
Removing intermediate container cb5d8388b402
Step 4 : RUN apt-key add /tmp/rabbitmq-signing-key-public.asc
 ---> Running in 80271b91c170
OK
 ---> b731bfc8085f
Removing intermediate container 80271b91c170
Step 5 : RUN echo "deb http://www.rabbitmq.com/debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list
 ---> Running in e13e4ca54531
 ---> 416ee2817824
Removing intermediate container e13e4ca54531
Step 6 : RUN apt-get -qq update > /dev/null
 ---> Running in dedd4a83548a
 ---> f89c5c5e12c8
Removing intermediate container dedd4a83548a
Step 7 : RUN apt-get -qq -y install rabbitmq-server > /dev/null
 ---> Running in e89f73400e08

Building from a GitHub Enterprise URL doesn't

However, I cannot reproduce the same behavior in our GitHub Enterprise URLs.

$ sudo docker build github.intuit.com/servicesplatform-docker/microservice-nodejs.git
FATA[0000] stat github.intuit.com/servicesplatform-docker/microservice-nodejs.git: no such file or directory

Also tried with git:// protocol, but I got a connection refused... Is this because of port numbers?

$ sudo docker build git://github.intuit.com/servicesplatform-docker/microservice-nodejs.gitFATA[0011] Error trying to use git: exit status 128 (Cloning into '/tmp/docker-build-git172970857'...
fatal: unable to connect to github.intuit.com:
github.intuit.com[0: 10.168.0.56]: errno=Connection refused
)

Also tried git+https:// 

$ sudo docker build git+https://git@github.intuit.com:servicesplatform-docker/microservice-nodejs.git#HEAD
FATA[0000] stat git+https://git@github.intuit.com:servicesplatform-docker/microservice-nodejs.git#HEAD: no such file or directory
duglin commented 9 years ago

as you've seen, accessing a git repo does work in general. In your particular case, are you sure that your daemon can reach your git repo? Try to create a container:

docker run -ti ubuntu bash

and in there try to access the URL to your repo - perhaps via curl and see if you get an error. My guess is that your daemon can't reach it and so it could be a network issue and not a docker issue.

marcellodesales commented 9 years ago

Hi @duglin

The connectivity worked for both servers, so the answer is yes, the daemon can access our GitHub repository (see below)... I did curl both and it is working...

What's Docker doing exactly with the command "docker build GIT_URL"? Is there an API call? Maybe that's what's failing? I can try to use wireshark to verify...

Ubuntu image

$ docker run -ti ubuntu bash
Unable to find image 'ubuntu:latest' locally
ubuntu:latest: The image you are pulling has been verified
27d47432a69b: Pull complete 
5f92234dcf1e: Pull complete 
51a9c7c1f8bb: Pull complete 
5ba9dab47459: Pull complete 
511136ea3c5a: Already exists 
Status: Downloaded newer image for ubuntu:latest
root@7cbc00bb39ff:/#

Installed curl

root@7cbc00bb39ff:/# apt-get install curl
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  ca-certificates krb5-locales libasn1-8-heimdal libcurl3 libgssapi-krb5-2
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhx509-5-heimdal libidn11 libk5crypto3 libkeyutils1
  libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2
  libroken18-heimdal librtmp0 libsasl2-2 libsasl2-modules libsasl2-modules-db
  libwind0-heimdal openssl
Suggested packages:
  krb5-doc krb5-user libsasl2-modules-otp libsasl2-modules-ldap
  libsasl2-modules-sql libsasl2-modules-gssapi-mit
  libsasl2-modules-gssapi-heimdal
The following NEW packages will be installed:
  ca-certificates curl krb5-locales libasn1-8-heimdal libcurl3
  libgssapi-krb5-2 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal
  libheimntlm0-heimdal libhx509-5-heimdal libidn11 libk5crypto3 libkeyutils1
  libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2
  libroken18-heimdal librtmp0 libsasl2-2 libsasl2-modules libsasl2-modules-db
  libwind0-heimdal openssl
0 upgraded, 25 newly installed, 0 to remove and 0 not upgraded.
Need to get 2668 kB of archives.
After this operation, 11.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main libroken18-heimdal amd64 1.6~git20131207+dfsg-1ubuntu1 [40.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty/main libasn1-8-heimdal amd64 1.6~git20131207+dfsg-1ubuntu1 [160 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty/main libkrb5support0 amd64 1.12+dfsg-2ubuntu4 [29.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ trusty/main libk5crypto3 amd64 1.12+dfsg-2ubuntu4 [79.5 kB]
...
...

Verified connectivity and it works

root@7cbc00bb39ff:/# curl -I https://github.com/mikaelhg/docker-rabbitmq
HTTP/1.1 200 OK
Server: GitHub.com
Date: Sun, 15 Feb 2015 19:33:41 GMT
Content-Type: text/html; charset=utf-8
Status: 200 OK
Content-Security-Policy: default-src *; script-src assets-cdn.github.com collector-cdn.github.com; object-src assets-cdn.github.com; style-src 'self' 'unsafe-inline' 'unsafe-eval' assets-cdn.github.com; img-src 'self' data: assets-cdn.github.com identicons.github.com www.google-analytics.com collector.githubapp.com *.githubusercontent.com *.gravatar.com *.wp.com; media-src 'none'; frame-src 'self' render.githubusercontent.com gist.github.com www.youtube.com player.vimeo.com checkout.paypal.com; font-src assets-cdn.github.com; connect-src 'self' ghconduit.com:25035 live.github.com wss://live.github.com uploads.github.com www.google-analytics.com s3.amazonaws.com
Cache-Control: no-cache
Vary: X-PJAX
X-UA-Compatible: IE=Edge,chrome=1
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Thu, 15-Feb-2035 19:33:41 GMT; secure; HttpOnly
Set-Cookie: _gh_sess=eyJzZXNzaW9uX2lkIjoiM2IwZjU4NGM2YmE2YWEwNmRkNTUzM2Q2OGYyN2I4OGYiLCJzcHlfcmVwbyI6Im1pa2FlbGhnL2RvY2tlci1yYWJiaXRtcSIsInNweV9yZXBvX2F0IjoxNDI0MDI4ODIxLCJfY3NyZl90b2tlbiI6InJGbzJVeW5qa1RaTXZWN01FejRqV3hnYUNUMnpYUWl5MEhrUlkrcEdaaWs9In0%3D--6dbd6b6641ea9ae94405da0cafe48b75cdb52a13; path=/; secure; HttpOnly
X-Request-Id: 081801b746264f7af5b01fec8b66986b
X-Runtime: 0.030593
X-Rack-Cache: miss
X-GitHub-Request-Id: C7108C1C:202B:2CEF8502:54E0F495
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Vary: Accept-Encoding
X-Served-By: 76f8aa18dab86a06db6e70a0421dc28c

root@7cbc00bb39ff:/# 
root@7cbc00bb39ff:/# curl -I https://github.intuit.com/servicesplatform-docker/microservice-nodejs
HTTP/1.1 200 OK
Server: GitHub.com
Date: Sun, 15 Feb 2015 19:33:31 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Status: 200 OK
Cache-Control: private, max-age=0, must-revalidate
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src *; script-src 'self'; style-src 'self' 'unsafe-inline' 'unsafe-eval'; object-src 'self'
Vary: X-PJAX
Set-Cookie: logged_in=no; domain=github.intuit.com; path=/; expires=Thu, 15-Feb-2035 19:33:31 GMT; HttpOnly
Set-Cookie: _fi_sess=sdsdsdsd; path=/; HttpOnly
ETag: "10385119958d7fecfdcfef98b6491f10"
Content-Length: 33224
X-GitHub-Request-Id: f6203281-2b71-40f0-a555-41cf7db09304
X-Content-Type-Options: nosniff

root@7cbc00bb39ff:/#
duglin commented 9 years ago

Do me a favor, try:

sudo docker build http://github.intuit.com/servicesplatform-docker/microservice-nodejs.git

and then with https.

duglin commented 9 years ago

And:

sudo docker build github.intuit.com/servicesplatform-docker/microservice-nodejs
duglin commented 9 years ago

And :-) from within the container, try curl on https://github.intuit.com/servicesplatform-docker/microservice-nodejs.git

marcellodesales commented 9 years ago

@duglin surprisingly, I might have forgotten to test it with just https (maybe I was too :zzz:)... It worked...!!!! :yum: Here are the tests for all the scenarios... Thanks a lot!

$  sudo docker build http://github.intuit.com/servicesplatform-docker/microservice-nodejs.git
[sudo] password for mdesales: 
FATA[0001] Error trying to use git: exit status 128 (Cloning into '/tmp/docker-build-git304606088'...
fatal: unable to access 'https://github.intuit.com/servicesplatform-docker/microservice-nodejs.git/': GnuTLS recv error (-9): A TLS packet with unexpected length was received.
) 
$ sudo docker build https://github.intuit.com/servicesplatform-docker/microservice-nodejs.git
Sending build context to Docker daemon 60.93 kB
Sending build context to Docker daemon 
Step 0 : FROM dockerhub.corp.intuit.net/intuit/nodejs
 ---> 1bf290eca8fa
Step 1 : MAINTAINER Marcello_deSales@intuit.com
 ---> Using cache
 ---> 8fd859f198a0
Step 2 : ONBUILD add package.json /tmp/package.json
 ---> Using cache
 ---> 28eccc129fe3
Step 3 : ONBUILD run cd /tmp && npm install && npm dedupe && npm prune --production && npm cache clean
 ---> Using cache
 ---> 85ef1bf8e13e
Step 4 : ONBUILD run mkdir -p /opt/app && cp -a /tmp/node_modules /opt/app/
 ---> Using cache
 ---> a1d822b1e115
Step 5 : ONBUILD workdir /opt/app
 ---> Using cache
 ---> 7546152fc470
Step 6 : ONBUILD add . /opt/app
 ---> Using cache
 ---> b448bd94fb7f
Step 7 : EXPOSE 3000
 ---> Using cache
 ---> e287cf19c73f
Step 8 : CMD npm start
 ---> Using cache
 ---> 5207903a7b7d
Successfully built 5207903a7b7d
$ sudo docker build github.intuit.com/servicesplatform-docker/microservice-nodejs.git
FATA[0000] stat github.intuit.com/servicesplatform-docker/microservice-nodejs.git: no such file or directory
marcellodesales commented 9 years ago

@duglin, Shouldn't Docker try to append "https" for GitHub Enterprise purposes?

That way, it would not confuse users because of the missing protocol, as it is supported by Docker with the public GitHub...

duglin commented 9 years ago

ok, here's what I think is happening.... when you tried

$ sudo docker build github.intuit.com/servicesplatform-docker/microservice-nodejs.git
FATA[0000] stat github.intuit.com/servicesplatform-docker/microservice-nodejs.git: no such file or directory

the logic in the CLI looked at your URL and determined it wasn't a git URL because it didn't start with http://, https://, git:// or git@ - so it treated it like a dir and that's why it says "stat" in the error - it couldn't find it on disk.

The problem with URLs w/o a transport is that its really hard to know if its a URL or just a path on disk that just happens to look like a URL. After all its valid to have a dir with that name.

marcellodesales commented 9 years ago

Got it... However, why does it work with the github.com URL? Is it treated differently?

duglin commented 9 years ago

@marcellodesales sadly yes. See https://github.com/docker/docker/blob/master/pkg/urlutil/git.go#L8 ok if we close this?

marcellodesales commented 9 years ago

@duglin... Ok... so nothing we can do... then close it... but I would expect it to be working nicely with any github the same way.

marcellodesales commented 7 years ago

Github Repositories with Dir Not working from Dockerfile

@duglin I tried using this feature again with directories. According to https://docs.docker.com/engine/reference/commandline/build/#git-repositories, we can use the following format for the master branch: url.git#:dir. 

/tmp/intuit-base on  master ⌚ 12:14:25
$ docker --version
Docker version 17.06.0-ce, build 02c1d87
FROM https://github.intuit.com/docker/intuit-base.git#:rhel7

Error

$ docker build --build-arg STASH_VERSION=3.43.1 -t idps-stash  .
Sending build context to Docker daemon   2.56kB
Step 1/12 : FROM https://github.intuit.com/docker/intuit-base.git#:rhel7
invalid reference format

Github Repo with Dockerfile in Directory

$ git clone https://github.intuit.com/docker/intuit-base
Cloning into 'intuit-base'...
remote: Counting objects: 61, done.
remote: Total 61 (delta 0), reused 0 (delta 0), pack-reused 60
Unpacking objects: 100% (61/61), done.

$ ls -la 
total 16
drwxr-xr-x   6 mdesales  wheel   204 Jul 12 11:37 .
drwxrwxrwt  26 root      wheel   884 Jul 12 12:11 ..
drwxr-xr-x  13 mdesales  wheel   442 Jul 12 11:37 .git
-rw-r--r--   1 mdesales  wheel  8138 Jul 12 11:37 README.md
drwxr-xr-x   4 mdesales  wheel   136 Jul 12 11:37 rhel6
drwxr-xr-x   4 mdesales  wheel   136 Jul 12 11:37 rhel7

$ ls -la /tmp/intuit-base/rhel7
total 8
drwxr-xr-x  4 mdesales  wheel  136 Jul 12 11:37 .
drwxr-xr-x  6 mdesales  wheel  204 Jul 12 11:37 ..
-rw-r--r--  1 mdesales  wheel  234 Jul 12 11:37 Dockerfile
drwxr-xr-x  3 mdesales  wheel  102 Jul 12 11:37 yum.repos.d

It works from Command-line

$ docker build https://github.intuit.com/docker/intuit-base.git\#:rhel7 -t rhel7
Sending build context to Docker daemon  3.584kB
Step 1/5 : FROM registry.access.redhat.com/rhel7/rhel
 ---> 3874a5cf3e5e
Step 2/5 : MAINTAINER Marcello_deSales@intuit.com
 ---> Using cache
 ---> 4116120bee5d
Step 3/5 : RUN yum remove -y subscription-manager
 ---> Using cache
 ---> 835cef929079
Step 4/5 : COPY ./yum.repos.d/INTU-LATEST.repo /etc/yum.repos.d/
 ---> c2406d333eb6
Removing intermediate container 8c93fb45ec69
Step 5/5 : VOLUME /etc/yum.repos.d/
 ---> Running in 3cbcddc8945f
 ---> 93a262daa838
Removing intermediate container 3cbcddc8945f
Successfully built 93a262daa838
Successfully tagged rhel7:latest
thaJeztah commented 7 years ago

FROM refers to an image not to a Dockerfile or build-context, so you cannot use FROM github.com/foo/bar

When running:

docker build https://github.intuit.com/docker/intuit-base.git\#:rhel7 -t rhel7

The following happens;

When using

FROM foo:bar

Docker pulls the image foo:bar from the registry, and executes the commands in the Dockerfile.

This is not a bug, so I'll close this, but feel free to comment after I closed

marcellodesales commented 7 years ago

@thaJeztah I thought that it would be the same scenario in Dockerfile, but it makes sense... Thank you!