search

gocd / gocd

GoCD - Continuous Delivery server main repository
https://www.gocd.org
Apache License 2.0
7.05k stars 973 forks source link

SSH GIT support (on remote GO Agent or GO Server) #951

Open toddb opened 9 years ago

toddb commented 9 years ago

Problem

Cannot connect to private git repository (eg BitBucket)

[go] Start updating files at revision 35e34....772457bccb from ssh://git@[myrepo].git
Cloning into '/home/go/pipelines/work-in-progress'...
STDERR: Host key verification failed.
STDERR: fatal: Could not read from remote repository.
STDERR: 
STDERR: Please make sure you have the correct access rights
STDERR: and the repository exists.
Failed to run git clone command

Situation

The necessary list (not wish list): support for SSH GIT is poor probably because it assumes git repositories are public and not private:

ajk8 commented 9 years ago

+1

kerrmarin commented 9 years ago

+1

nhudacin commented 9 years ago

+1

scaryclam commented 8 years ago

+1

Ehekatl commented 8 years ago

+1

pejvan commented 8 years ago

+1

robearlam commented 8 years ago

+1

zstarer commented 8 years ago

+1

vspiewak commented 8 years ago

+1

yukatan commented 8 years ago

+1

TrentBartlem commented 8 years ago

+1

zhmocean commented 8 years ago

+1

brettcave commented 8 years ago

You could generate a key pair on every server and agent, as well as accept the fingerprint... or you could provision the agents / server config files using shared configs (e.g. chef / puppet / etc).

Add the following to ~go/.ssh/config:

StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Generate an RSA once on an administrator host. Upload the public key to the Git repo and add the private key to each host, under ~go/.ssh/id_rsa. e.g. with Chef:

file "#{go_home_dir}/.ssh/id_rsa" do
  user node[:go][:user]
  mode 0600
  content my_encrypted_databag_items["git_ssh_private_key"]
done
ketan commented 8 years ago

Here is what I think could be the solution —

the <git /> <hg/> and <svn/> materials would have an additional attribute ssh_key. Users would need to provide an ssh key without a passphrase (to start with with).

The agents would then execute a git operation as follows (need to investigate if svn and hg support this directive) —

$ echo ${SSH_KEY} > ${TMPDIR}/ssh_key # or use ssh-agent if appropriate
$ echo 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${TMPDIR}/ssh_key $*' > ${TMPDIR}/ssh
$ chmod u+x ${TMPDIR}/ssh
$ GIT_SSH=${TMPDIR}/ssh git ...whatever...
$ rm -rf ${TMPDIR}/ssh_key

Version 2.3 of git supports GIT_SSH_COMMAND that can reduce the number of commands that we run —

$ echo ${SSH_KEY} > ${TMPDIR}/ssh_key # or use ssh-agent if appropriate
$ GIT_SSH_COMMAND='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i ${TMPDIR}/ssh_key' git ...whatever...
$ rm -rf ${TMPDIR}/ssh_key
zeusro commented 8 years ago

Fine,what should I do in the Windows Server's Go Server?

KIVagant commented 8 years ago

If I can't use git with ssh keys in gocd out the box, then I can't understand, how to use gocd at all? UP: OK, I found the solution here. First of all the ssh keys should be copied to /var/go/.ssh/ with chown -R go:go /var/go/.ssh/; chmod -R 400 /var/go/.ssh/

TrentBartlem commented 8 years ago

You need to create ssh keys for the server and all the agents and register them with the git server.

There's no way to set ssh keys in the UI, or to push them to agents.

FredrikWendt commented 8 years ago

Well, GoCD currently doesn't manage keys for you, but you can still put them in the default directory so the SSH binary can pick them up. We build a Docker image for the GoCD agents, ready baked with some tools, prepopulated .m2/repository, SSH key, ...

/ Fredrik Den 4 feb. 2016 12:01 em skrev "Trent Bartlem" notifications@github.com:

You need to create ssh keys for the server and all the agents and register them with the git server.

There's no way to set ssh keys in the UI, or to push them to agents.

— Reply to this email directly or view it on GitHub https://github.com/gocd/gocd/issues/951#issuecomment-179771720.

steimntz commented 8 years ago

+1 I agree manager keys in a Unix environment isn't such a hard thing however I think GoCD can be more friendly with this functionality.

Moeser commented 8 years ago

I'm surprised this is still an issue in GoCD. If you're running GoCD from a docker container then managing the ssh private keys un-encrypted is a security risk.

ashutoshraina commented 8 years ago

+1

ketan commented 7 years ago

To further my comment above, there are 3 possible ways to allow specify an SSH key for each git invocation —

ssh-agent

ssh-agent sh -c 'ssh-add ~/.ssh/id_rsa; git fetch user@host'

Pros:

Cons:

GIT_SSH_COMMAND

GIT_SSH_COMMAND='ssh -i ~/.ssh/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' git clone user@host

Cons:

Pros:

GIT_SSH

$ echo 'ssh -i ~/.ssh/id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $*' > ssh
$ chmod +x ssh
$ GIT_SSH='./ssh' git clone user@host

Cons:

Pros:

ketan commented 7 years ago

To add — I'm proposing adding an attribute (or CDATA element) to the <git/> tag (to begin with) we may later consider sharing ssh keys via some other means.

What this means is that the config XML will look like, the actual key content will be encrypted, but is displayed in the clear as an example.

  <git url="git@git.example.com:Acme/FooBarWidgets.git" 
     sshKey="
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
"
     />

Or a better form using CDATA —

<git url="git@git.example.com:Acme/FooBarWidgets.git">
  <sshKey>
    <![CDATA[
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----
    ]]>
  </sshKey>
</git>

In the future we may consider adding a top level <sshKeys/> tag —

<sshKeys>
  <sshKey id="bob's private key">
    <![CDATA[
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----
    ]]>
  </sshKey>
  <sshKey id="foobar shared key">
    <![CDATA[
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----
    ]]>
  </sshKey>
</sshKeys>

<git url="git@github.com/bob/bob.github.io" sshKeyId="bob's private key" />

<git url="git@github.com/acme-corp/foobar.git" sshKeyId="foobar shared key" />

WDYT?

tomzo commented 7 years ago

Some option which would allow to not distribute ssh keys to go agents at all is by leveraging ssh agent forwarding.

FredrikWendt commented 7 years ago

1) Any solution that is Git only would be local and a laugh at any org using Mercurial or some other DVCS (anything Launchpad.net accepts for instance) 2) Central management, top level sshKeys element, with pipeline tags referencing keys would scale better. 3) Central management, with per job created ssh-agent instance, forwarded to the agent sounds preferable from a security point of view.

mrwiora commented 7 years ago

temporary fix for current version of go.cd (16.5)

mkdir -p /var/go/.ssh;
# insert your private key to /var/go/.ssh/id_rsa
ssh-keyscan gitserver.tld > /var/go/.ssh/known_hosts # replace gitserver.tld with your git destination
chown -R go:go /var/go/.ssh/; chmod -R 400 /var/go/.ssh/id_rsa
cdluv commented 7 years ago

Assuming you've created your key and it's in ~/.ssh/id_rsa with the correct permissions, you can add the following to the end of /etc/profile to ensure the ssh agent always runs in go server and go agents. It also ensures the GitHub key is registered to the agent.

There may be some security implications with this, but it's got me moving forward.


# Start ssh-agent for user if not already started
ps -u $(whoami) | grep ssh-agent || eval $(ssh-agent)

# Register id_rsa with ssh-agent to automate Git private repo Git authentication
echo | ssh-add ~/.ssh/id_rsa

Then naturally, restart the go-agent and go-server services so they inherit the ssh-agent's session.

andrask commented 7 years ago

+1

flypenguin commented 7 years ago

I hope you don't plan to rely on the host machine for managing git ssh keys. I run the agent and the server in a container, and when I configure the host that gives me ... nothing. also I really dont want to derive from the gocd/gocd-server image myself to add some ssh keys just to access my repos.

chnliyong commented 7 years ago

+1

wesselOC commented 7 years ago

+1

lucciano commented 7 years ago

+1

mribichich commented 7 years ago

+1

Frankstar commented 7 years ago

+1 - what a deal breaker - sad

flypenguin commented 7 years ago

is there any timeline on this? I would love to try goCD to replace a our TeamCity instance, unfortunately this is a hard deal breaker.

Frankstar commented 7 years ago

interresting, goCD had so many problems and missing features i gave up on it and switched to TeamCity (came orginally from Bamboo) - Why you want replace TC? version 10 looks promising

flypenguin commented 7 years ago

funny :) .

to answer: we don't use about 99% of TCs feature set, the only thing TC does is running a 400+ lines shell build script which actually handles all our builds including all weird corner cases by using a general docker build container approach combined with checked-in build scripts.

so paying for TC this seems quite unnecessary. we would go for jenkins, but then we'd have the same overblown solution for a somewhat simple problem. maybe gocd is the wrong solution, but it looks right ;) ... except for the missing SSH access, that's just ... not.

EugenMayer commented 7 years ago

Coming from Bamboo, we evaluated quiet some, also GoCD. GoCD is pretty solid and has its SOP ( Pipelines are definitely the best on the market ) and there overall concept with materials, pipelines, stages and so on seems to be well thought and very flexible. The templates are also a SOP, afaik there is no other system offering that

In the end, we compared ths agains GitLab CI and the latter won on every single field except pipelines. GoCD is just not there yet, they seems to just been waking up.

From Elastic Agents ( its ok, but its seems like alpha, a lot of things are yet not possible, very complicated or conceptually just not 'there' ), same goes for overall "in container builds", same goes for CI-tasks in files, GitLab is far superior. Setting up a Elastic-Agent compat Image to let GoCD build in is PITA, overcomplicated an by far not best practise in any way, while gitlab is non-intrusive, well done and easy as hell.

Also for how they encapsulated "tasks/jobs" and how to run them in "shell, remote shell, container" - in GitLab, you select a type and can run the same set of commands. On most other systems, your executes do vary from the type of "shell, remote shell or container", so you cannot swap.

Overall, there is a pro/con as always and i am sure GoCD will rise in the future. For us at least it was (yet) not the choice to make.

Frankstar commented 7 years ago

well - its a bad thing to hijack this issue for that but - since you only have 1 Build in TC (im i right?) you can use the free Professionell Licence - so pretty much no costs.

i have similiar problem like you and thought the same about goCD. but things like ssh-key management need to be in by default.

take a look @ bamboo by atlassian - i think this might be the perfect solution for you.

flypenguin commented 7 years ago

@Frankstar well, not quite at no cost ;) . But I won't go into more details. Bottom line is - we can't evaluate goCD because of this SSH thing.

@EugenMayer still thanks for the info! let me get back to you by mail with a followup question, if you don't mind :)

arvindsv commented 7 years ago

I appreciate this latter part of this discussion. It has useful information, rather than just +1s or -1s. :) @ketan is trying out a way of managing keys using a new material (starting off with GitHub for now, and if it works, maybe we can move it into git, mercurial, etc). He's been finding out that handling SSH keys across different OSes and across different kinds of SCM materials is not at all straightforward. I'll let him elaborate if he wants to.

As I've said before, it's an open-source project. I personally appreciate a good discussion or a pull request over useless comments or +1s. That's why I chimed in to mention this part of the discussion. If any of you are good developers and want to contribute, you'll always be welcome. Unfortunately, we can't always get to everything. I know this is important.

I feel some of this is more a problem of SSH key distribution than SSH key support.

FredrikWendt commented 7 years ago

I feel some of this is more a problem of SSH key distribution than SSH key support.

What? Have you even tried setting this up with Infrastructure as Code? I find your comment ignorant, almost offending. What is GoCD's proposed way to get to auditable source code access? Username and password? Several options exists (see top post) that allows avoiding SSH key distribution completely. But yes, the current way one needs to workaround this issue is to rebuild BOTH the server and agent image (docker, AWS, ...) whenever a new SSH key needs to be used, restart BOTH the server and all agents. So yes, right now this workaround is about SSH key distribution ...

@ketan is trying out [...] handling SSH keys across different OSes [...] different kinds of SCM materials [...]

I've heard of ThoughtWorkers promoting iterative, incremental delivery. Some call this having an agile mindset: to deliver value early and often. I love the product, but it's amazing how much "big batches" and how few things are put on display via some early access mechanism ("secret" but well-known feature flags with notes like "this may change, so try it out and give us feedback").

This issue - #951 - is about centrally managing SSH keys, and distributing them, as needed, to agents fetching GIT materials. Your comment refers to a bigger scope than what's asked for (the general SSH for auth is in #430). You may decide that's what's needed, but as of for several months (this general need was registered in Aug 2014, in #430) - this issue is (obviously IMHO) blocking market adoption.

It has useful information, rather than just +1s or -1s. :)

If you don't care about us providing feedback/voting on what we think is important, your (paying, advocating, contributing) users, ... meh. >:-(

PS. Sorry if I'm ranting too much here, but the above comment I more or less read as "go ahead and fix yourself, or please be silent", ie nothing I find very useful, no new info. "This is not on the roadmap for 2017" is useful, and also "we have 50 other features we're prioritizing above this right now, and we spit out 5 a month or so". Not expecting promises.

arvindsv commented 7 years ago

Hi @FredrikWendt: I'm in the middle of something, so this might not be very comprehensive. But, my response was not meant as "fix it yourself or please be silent". I get annoyed with +1s, because they're saying this is important (sure), but not really adding much else. I like discussions, as I said. GitHub has a better way of marking that.

My point about SSH key distribution is that: This is about distributing SSH keys from server to agent and putting it in the right place. Beyond that ssh agents and ssh support built into SCMs such as git (git://), mercurial (ssh://) and svn (svn+ssh://) and so, this is about distribution of keys from the server to the agent is what I said. I stand by that.

However, it has a slight problem that any task can access it as well and potentially cause problems. The change that @ketan is trying is trying is to prevent that key from being accessible by tasks.

I don't really want to respond to the feature flags part and exposing it. It gets done as and when possible though we might not always make it very easy to try things out. I'm proud of contributions and changes made to the core and plugins and everything, both by people on the core team and from people outside. Of course, some people won't be happy with the amount of progress and/or the speed of it. I understand not everyone can be happy with everything, and that's fine by me. I've figured out that trying to please everyone is useless.

Finally, I've talked to you before and I know you're very rational and nice to talk to and not at all like it's coming across in this comment. You've always helped by giving critical and encouraging feedback in a proper tone and I'll remember that.

flypenguin commented 7 years ago

afraid to ask - but: progress, anyone?

schlomo commented 7 years ago

I use this hack as a workaround for SSH keys in Go CD: https://github.com/schlomo/ssh-url-with-ssh-key

prayagupa commented 7 years ago

weird that I logged in as go user and creates public private keys, updated permission as well.

$ cat /etc/passwd | grep go
go:x:995:992:GoCD User:/var/go:/bin/bash

sudo su go
bash-4.2$ ls -l ~/.ssh
ls: cannot access /var/go/.ssh/gitkey_rsa: Permission denied
ls: cannot access /var/go/.ssh/gitkey_rsa.pub: Permission denied
ls: cannot access /var/go/.ssh/known_hosts: Permission denied
total 0
-????????? ? ? ? ?            ? gitkey_rsa
-????????? ? ? ? ?            ? gitkey_rsa.pub
-????????? ? ? ? ?            ? known_hosts

from root user, sudo chown -R go:go /var/go/.ssh/; chmod -R 400 /var/go/.ssh/ grants permission to ~/.ssh.

[root@ip-172-23-38-167 ~]# ls -l /var/go/.ssh
total 12
-r--------. 1 go go 1766 May 20 19:45 gitkey_rsa
-r--------. 1 go go  386 May 20 19:45 gitkey_rsa.pub
-r--------. 1 go go  413 May 20 19:46 known_hosts

Then loaded ~/.ssh/gitket_rsa, but can not connect to git server.

eval "$(ssh-agent)"

ssh -vT git@git.mycompany.net
oppianmatt commented 6 years ago

gocd needs to manage ssh keys that way a plugin like the github oauth plugin could create deploy keys when you give it access to your repo

There should be no need to distribute ssh keys to agents. This problem has been solved already using stuff like SSH_AUTH_SOCK. I can ssh into one machine and use the keys from my main machine when I ssh out without needing to copy keys.

If gocd was the central store and forwarded the keys using existing ssh functionality this would solve all those woes. Here is an example of how to use ssh forwarding https://developer.github.com/v3/guides/using-ssh-agent-forwarding/

Right now all the solutions consist of either turning of known hosts checking (bad), or baking a key into the docker and redeploying. It's basically adding state into a docker image which should be stateless. The agents shouldn't need to have ssh key as this is more config to manage and means agents aren't flexible. Means we have to currently create custom agents with state baked in. If you rotate/change your key you will need to redeploy ALL your agents with the new key. Also your key is baked into the agents which is a security risk. If an agent is compromised now your code is too. If you didn't use a deploy key but a normal github ssh key, they now have write access to all your secret sauce.

If the agent didn't have the key and used ssh key forwarding from the master that would be more secure and easier to manage. Agents would be just dumb agents then and easy to setup.

FredrikWendt commented 6 years ago

@oppianmatt Yes, ssh agent forwarding would be great, but there are some considerations making it a tad more complex:

One could create one key agent per key on the server (or per combination of keys, as needed by jobs), and tunnel the respective key agent out to gocd agents, per job.

CharlesWinter commented 6 years ago

Just thought I would leave my two cents here on how I managed this problem.

My problem wasn't exactly with using ssh keys, but more related to how to secure access to a private github account in general (Apologies to those not using Github, this solution probably won't help you)

I began use personal access tokens , however this was simply entered as a material in plain text, meaning everyone was able to see each others personal access tokens. After digging deep on the the internet, I found that entering the personal access token in the format

https://x-access-token:<personal_token>@github.com/<username>/<repo>.git

would result in the token being hidden from users of the GoCD UI platform.

I would advise GoCD that this 'phenomenon' is extremely poorly documented, and I found the solution on a thread from 2014 here. https://groups.google.com/forum/#!topic/go-cd/4RPtwXSvrgE

remy-tiitre commented 6 years ago

I agree that it's not very intuitive but solution that works for us with Docker Swarm: We start our GoCD agents as Swarm Services and use Docker Secrets. Extract from docker-compose.yml:

    secrets:
      - source: git_access.key.v1
        target: git_access.key
        uid: '1000'
        gid: '1000'
        mode: 0600

In our GoCD agent Dockerfile we copy ssh config into container: COPY ssh_config /home/go/.ssh/config and set correct permissions for it. and this config has the following settings:

HOST some.domain.name.for.repo
    IdentityFile /run/secrets/git_access.key
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null
    IdentitiesOnly yes
    BatchMode yes

If you have a lot of different keys this might get out of hands quick, or if you connect to different repo servers. But right now its perfect as rotating keys can be done by someone who has no knowledge of GoCD.