Git Clone and Checkout failing

[dc00p]

I am having trouble getting the Docker Runner plugin working.

I am logged in through BitBucket.

When I use the simple runner the project builds fine.

When I try and use the Docker Runner I get the following output:

git clone --recursive git@bitbucket.com:[repository] . --branch master
Cloning into '.'...
$ git checkout --quiet --force master bitbucket 13ms -1
fatal: Not a git repository (or any of the parent directories): .git

An unexpected error occurred while executing this job. Please report at https://github.com/Strider-CD/strider/issues

Checkout failed with code 128

When attempting to do a manual clone from the strider/strider-docker-slave container itself I get the following error:

Permission denied (publickey).
fatal: Could not read from remote repository.

I'm not sure if this is related, or a coincidence.

How are the credentials passed from the machine running Strider-CD to the Docker Runner anyway?

Any help would be appreciated.

[skruppy]

Same problem here. What makes me wonder is the fact, that it seems to have never worked before?!? I randomly picked some other git tags and tried the master branch of strider itself and strider-docker-runner. Even with an updated docker image using Ubuntu 5.10 the problems stayed the same.

Maybe its a problem with strider-docker-slave? (but there hasn't been any updates recently)

@dc00p The ssh key is setup in the docker image right before the git command is run. When you start your own docker instance, then there is no ssh key. Thats expected behavior.

[knownasilya]

The best person to give you a status would be @keyvanfatehi, or maybe @davemackintosh (I think he worked with the runner to an extent).

[davemackintosh]

Correct me if I'm wrong @knownasilya @keyvanfatehi. The runner should clone to repo on the Strider machine, not the container and then git zip and copy the zip over to the target container for running, etc.

The error 128 means a permissions issue, while the key might be correct/added, the container either doesn't have the key OR the commands are being run as a different user to the one you expect.

Can you add RUN echo $(whoami) to your docker file and see if the user is what you expect?

[xdissent]

We're using the docker runner with great success, but with github rather than bitbucket. It does currently run the clone on the actual container. I think that's the correct behavior though, since it's part of the prepare phase which certainly needs to run inside the container. It's likely that the bitbucket plugin is just confusing a local path with a path on the container. I'd try setting DEBUG='strider-docker-runner:*,strider-bitbucket:*' as a first step. I started hacking on the docker runner last night to fix a similar issue with the node plugin. I'll drop a note here in a little bit when I push that branch - perhaps it'll fix this too.

[davemackintosh]

Agreed, that's the correct behaviour. Please let us know your findings :) I haven't had a lot of time to stay familiar with this plugin so all refreshers are welcome!

[xdissent]

The bitbucket and github plugins both use strider-git for the actual cloning. The difference is that github uses https urls with an authentication token, whereas bitbucket uses ssh, registering the project's pubkey via api. Here's where it gets funny.

The git plugin runs the git command directly, using the correct context.cmd() method as provided by the runner (which pipes the commands to the docker slave) - except when the repo uses ssh (like bitbucket). In that case, it uses Gitane for handling the key setup and actual clone. It turns out Gitane let's you pass in a custom spawn() function, so it could technically work with the docker runner, which provides a docker-compatible spawn() function for just such an occassion. However, as seen here, the git plugin doesn't pass the correct spawn function to Gitane. Assuming I don't have my contexts all mixed up (there are a lot of different configs and contexts in the mix!), then it should be context.spawn rather than context.runCmd.

But even then, Gitane makes direct calls to fs to setup the keys, rather than using commands. That means it's copying keys to your local host, then trying to run the clone command inside the container. Somehow it thinks the clone succeeds, but I suspect that's another bug somewhere.

Direct fs access is a huge issue, present in most of the plugins I've looked at. I basically got lucky with the github plugin. The whole host/container path problem is also real, but direct fs access is more prevalent.

I've refactored the docker runner to fix the path/cwd/env related issues, but many plugins would need to be updated before they'd actually work.

As an aside, I keep seeing a lot of gymnastics to support Windows... how serious are we about this? If we were to update plugins to be command-driven-only, things would get hairy pretty quickly. Also, the n module, a dependency for strider-node, which is a core plugin, doesn't even install on Windows. Just sayin :wink:

[knownasilya]

As an aside, I keep seeing a lot of gymnastics to support Windows... how serious are we about this?

I don't think we are too serious about windows support, since most Strider instances need to be hosted and most nodejs based hosting platforms are Linux based.

Direct fs access is a huge issue

Can you expound on this topic a little?

[xdissent]

@knownasilya Sure. Take the node plugin for example. It checks to see if a package.json file is actually present before running npm install. To do that, it uses fs.existsSync. The problem is that it's checking on the local host, not the actual container which contains the repo, so it always fails. The docker container only understands shell commands, so it should really be doing something like sh -c '[ -f package.json ]' and evaluating the exit code.

[sebas5384]

It seems to be happening only with private repos.

This issue would need some refactoring on the strider-git module? something like creating a hand crafted compatible git node lib instead of using gitane.

Whats could be the path to fix this issue?

[knownasilya]

@sebas5384 The people that originally created gitane (and are more knowledgeable on the topic) are scarce around these parts @niallo @jaredly. So your guess is as good as mine :+1:

Throw ideas out, and we can discuss.

[niallo]

I wrote Gitane a while back, it's very simple. I didn't think much about Windows support.

Git has improved in the past 3 years. There is even a pure-JS implementation of Git :-)

I'm happy to accept PRs on the repo, but it might make sense to use one of these:

• NodeGit https://github.com/nodegit/nodegit (seems to have Windows support)
• JSGit https://github.com/creationix/js-git (native javascript - should work on any platform)

On Tue, Dec 8, 2015 at 6:38 AM, Ilya Radchenko notifications@github.com wrote:

@sebas5384 https://github.com/sebas5384 The people that originally created gitane (and are more knowledgeable on the topic) are scarce around these parts @niallo https://github.com/niallo @jaredly https://github.com/jaredly. So your guess is as good as mine [image: :+1:]

Throw ideas out, and we can discuss.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider-docker-runner/issues/36#issuecomment-162901426 .

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins