"Machine user" fails as a deployment mechanism for private submodules

[TeaSeaLancs]

I'm attempting to do an automated build which uses private git submodules.

I've been following the guide at https://github.com/programmerq/docker/blob/master/docs/sources/docker-hub/builds.md but it just doesn't work. Every build fails with the error:

Building in Docker Cloud's infrastructure...
Starting to clone
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '***.**.***.***' to the list of known hosts.
Submodule 'common' (https://github.com/***/common) registered for path 'common'
Cloning into 'common'...
fatal: could not read Username for 'https://github.com': No such device or address
Clone of 'https://github.com/***/docker-common' into submodule path 'common' failed
please ensure the correct public key is added to the list of trusted keys for this repository and the remote branch exists. (1)

[pchico83]

Hi @TeaSeaLancs We are working on updating the docs, you can configure an autobuild with private submodules by doing:

• Create a github machine user.
• Create a pair of private/public ssh keys
• Upload the public SSH key to the github machine user account.
• Add and environment variable in your autobuild called SSH_PRIVATE whose value is the private ssh key.

[TeaSeaLancs]

Hi @pchico83 thanks for the update.

Just to clarify (Although i'm sure i'll find this out during configuring this), this environment variable should be put into the dockerfile as an ENV statement?

[pchico83]

@TeaSeaLancs You don't need it in the dockefile, you need to add it in the Docker Cloud UI, as part of your autobuild configuration.

[TeaSeaLancs]

And therein lies the issue: I was using Docker Hub autobuilds.

I've just switched to Docker Cloud so I will give that a shot and let you know.

[pchico83]

This could be interesting to you, talks about the future direction of Docker Cloud and Docker Hub: https://forums.docker.com/t/future-updates-on-docker-hub-docker-cloud/20498

[TeaSeaLancs]

Thanks for that, it looks like good reading.

Having followed your instructions above, unfortunately the build still fails with the same error message. I have ensured that the public key is attached to the machine user, the machine user has read & write access to the private repo docker-common (Plus the actual private repo that we're building!), and added the private key as SSH_PRIVATE under the autobuild environment settings, and it still doesn't work.

Is there anything I could be doing to help figure out what's going on?

[TeaSeaLancs]

This is a screenshot of how it's set up, this all seems correct?

[TeaSeaLancs]

Also, does it matter if the machine user is the user that is linked to Github in Docker Cloud? (It currently is)

[pchico83]

@TeaSeaLancs No, it doesn't. But you will still need to add the SSH_PRIVATE env variable. This is because the same deploy key (which is what Docker Cloud use to clone repositories) can be only added to a single repo in github. The only way to clone several repositories using SSH is a user SSH key.

[TeaSeaLancs]

Ok. I did that, as shown in the screenshot above, and it still doesn't work, so i'm at a loss as to what to do :)

[pchico83]

@TeaSeaLancs From the error, looks like your git private submodules are using https instead of ssh.

[TeaSeaLancs]

And that could explain a few things! Thanks for bearing with me on this, this could be a massive facepalm on my side here.

[pchico83]

I opened an internal issue to document all this. It is a recur-rely question how to set up private submodules, and we don't have it documented anywhere.

[TeaSeaLancs]

Yeah I think mentioning the SSH thing in setting up the submodules could be a good idea as well, practically everyone just says to use the https:// format when adding submodules when you google about it.

I swapped the format to be git@github.com:organisation/repo in the submodules and now it's working great, thanks so much for the help.

[TeaSeaLancs]

One final question, is the SSH_PRIVATE mechanism exclusively for docker cloud, or is it possible to reuse this for manual builds?

[pchico83]

By "manual" do you mean for local builds? As long as you are using SSH keys to clone git repos in your local environment, it will work locally too. In a nutshell, Docker Cloud uses SSH_PRIVATE to clone your git repo and from there, it executes a docker build command the same way it is executed locally.

[TeaSeaLancs]

Ah yes that makes sense.

I was thinking of a slightly different usecase: We also have git private submodules which are used as part of an npm install during the build process, so we have to be able to get SSH keys available there as well. I was wondering if there was any Docker best-practice regarding how to do that as well.

[pchico83]

You could use the instruction ARG in your dockerfile to pass the SSH keys, but this is not recommended since the SSH key will be kept in the layers of your image. For these cases, I recommend a two-phase build. You can generate all the content (clone every git module) before executing docker build and use the ADD instruction to inject the content on your image without needing credentials or SSH keys as part of your build process. In Docker Cloud, you can make use of hooks to accomplish these scenarios: https://docs.docker.com/docker-cloud/builds/advanced/#/custom-build-phase-hooks

[TeaSeaLancs]

That's excellent advice, thanks so much!

[vhosakot]

I saw the same error when I triggered automated build for my private GitHub repo.

Building in Docker Cloud's infrastructure...
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

please ensure the correct public key is added to the list of trusted keys for this repository (128)

Adding the deploy Key in “Build Settings” in Docker Hub to my GitHub account at https://github.com/settings/keys resolved this issue for me and the automated build passed for my private GitHub repo.

[kaji-bikash]

I am sure the participants are in the position to answer correctly what works and what does not.

We are trying to build a workflow that works for our Ruby projects(mostly Rails and Sinatra). In ruby projects, bundler is a defacto standard to gather and maintain application dependency. Every project would have Gemfile. Bundler takes Gemfile to create a snapshot of the project with all the dependencies and that too without any conflict.

One of the very common cases is as such that Gemfile would have many private ruby gem(library) as dependencies for the project and those are typically stored in GitHub private repository. Sensing the analogy of git private submodules and following documentation around git private submodule for DockerHub/ documentation on Docker Cloud, machine user was created. As per the dockerhub documentation, the automated build deploy key was added to machine-user as SSH key and that machine-user was then given access to all the private repositories as required by the Gemfile and in order for the bundler to complete the dependency pull. Also as per docker cloud documentation, private key of public-private key was exported to SSH_PRIVATE environment varible and public half to the machine user SSH key in GitHub. But it's not working.

It always failed with the following

Dockerfile

.......redacted
ENV APPROOT ${HOME}/service
ENV BUNDLE_PATH /var/bundle

WORKDIR ${APPROOT}

#Fixing "GitHub Host key verification failed". It comes because of interactive screen on the command line
#Other option would be to turn off HostsKeyCheck which is dangerous and not recommended.
#RUN git config --global url."https://github.com/".insteadOf "git@github.com:" && \
RUN ssh-keyscan github.com >> ${HOME}/.ssh/known_hosts 
#&& ssh-keygen -f ${HOME}/.ssh/id_rsa -t rsa -N ''

ADD . ${APPROOT}

RUN gem install bundler

RUN echo ${SSH_PRIVATE}

RUN bundle install && ./configure

What is wrong here? @pchico83 /cc @TeaSeaLancs

We have subscription for DockerHub but looks like https://cloud.docker.com also gives access to same repositories but with some added/removed features. This is also very confusing.

[bgrande12]

@kajisaap Did you ever figure out a solution? I'm experiencing the same kind of issue

[kaji-bikash]

Sorry, we never could figure out and Docker community couldn't also help.

[github-actions[bot]]

We are clearing up our old issues and your ticket has been open for one year with no activity. Remove stale label or comment or this will be closed in 15 days.