search

JeanRev / TeamcityDockerCloudPlugin

Docker Cloud Plugin for the TeamCity build server
Apache License 2.0
33 stars 10 forks source link

Client-side checkout not working #41

Closed Vampire closed 6 years ago

Vampire commented 6 years ago

I hope I'm not wrong here and bother you for nothing. I just found your plugin and wanted to give it a try. Unfortunately, I cannot get it working properly. When I start a build on the Docker Cloud Agent, the client-side checkout does not work. (And seemingly some other stuff)

In the log I see

[2018-07-29 15:57:39,436]   WARN - l.ssl.SslAgentLifeCycleAdapter - Cannot find SSL certificates, response: HTTP/1.1 403 Forbidden
[2018-07-29 15:57:39,939]   WARN - uildServer.ssh.SshKeyRetriever - Cannot find SSH key javacord_vampire_deploy_key in context vcsRoot:46, response: HTTP/1.1 403 Forbidden
[2018-07-29 15:57:41,384]   WARN - l.patch.AbstractSourcesUpdater - Error while checkout on agent: '/usr/bin/git -c credential.helper= fetch --progress origin +refs/heads/v_3:refs/heads/v_3' command failed.
exit code: 128
stderr: [15:57:40.107] INFO SSH command to run: git-upload-pack 'Vampire/Javacord.git'
[15:57:40.169] INFO Connecting to github.com port 22
[15:57:40.279] INFO Connection established
[15:57:40.376] INFO Remote version string: SSH-2.0-libssh_0.7.0
[15:57:40.377] INFO Local version string: SSH-2.0-TeamCity-Agent-2018.1-JSCH-0.1.53
[15:57:40.377] INFO CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[15:57:40.884] INFO CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
[15:57:40.952] INFO CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
[15:57:40.955] INFO SSH_MSG_KEXINIT sent
[15:57:40.955] INFO SSH_MSG_KEXINIT received
[15:57:40.955] INFO kex: server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
[15:57:40.955] INFO kex: server: ssh-dss,ssh-rsa
[15:57:40.955] INFO kex: server: chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc
[15:57:40.955] INFO kex: server: chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc
[15:57:40.955] INFO kex: server: hmac-sha2-256,hmac-sha2-512,hmac-sha1
[15:57:40.955] INFO kex: server: hmac-sha2-256,hmac-sha2-512,hmac-sha1
[15:57:40.955] INFO kex: server: none,zlib,zlib@openssh.com
[15:57:40.955] INFO kex: server: none,zlib,zlib@openssh.com
[15:57:40.955] INFO kex: server: 
[15:57:40.955] INFO kex: server: 
[15:57:40.955] INFO kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
[15:57:40.955] INFO kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
[15:57:40.955] INFO kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
[15:57:40.955] INFO kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
[15:57:40.955] INFO kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[15:57:40.955] INFO kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[15:57:40.955] INFO kex: client: none
[15:57:40.955] INFO kex: client: none
[15:57:40.955] INFO kex: client: 
[15:57:40.955] INFO kex: client: 
[15:57:40.955] INFO kex: server->client aes128-ctr hmac-sha1 none
[15:57:40.955] INFO kex: client->server aes128-ctr hmac-sha1 none
[15:57:40.957] INFO SSH_MSG_KEX_ECDH_INIT sent
[15:57:40.957] INFO expecting SSH_MSG_KEX_ECDH_REPLY
[15:57:41.070] INFO ssh_rsa_verify: signature true
[15:57:41.079] WARN Permanently added 'github.com' (RSA) to the list of known hosts.
[15:57:41.080] INFO SSH_MSG_NEWKEYS sent
[15:57:41.080] INFO SSH_MSG_NEWKEYS received
[15:57:41.087] INFO SSH_MSG_SERVICE_REQUEST sent
[15:57:41.182] INFO SSH_MSG_SERVICE_ACCEPT received
[15:57:41.278] INFO Authentications that can continue: publickey,keyboard-interactive,password
[15:57:41.278] INFO Next authentication method: publickey
[15:57:41.279] INFO Disconnecting from github.com port 22
Auth fail
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
jetbrains.buildServer.vcs.VcsException: '/usr/bin/git -c credential.helper= fetch --progress origin +refs/heads/v_3:refs/heads/v_3' command failed.
exit code: 128
stderr: [15:57:40.107] INFO SSH command to run: git-upload-pack 'Vampire/Javacord.git'
[15:57:40.169] INFO Connecting to github.com port 22
[15:57:40.279] INFO Connection established
[15:57:40.376] INFO Remote version string: SSH-2.0-libssh_0.7.0
[15:57:40.377] INFO Local version string: SSH-2.0-TeamCity-Agent-2018.1-JSCH-0.1.53
[15:57:40.377] INFO CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[15:57:40.884] INFO CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
[15:57:40.952] INFO CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
[15:57:40.955] INFO SSH_MSG_KEXINIT sent
[15:57:40.955] INFO SSH_MSG_KEXINIT received
[15:57:40.955] INFO kex: server: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
[15:57:40.955] INFO kex: server: ssh-dss,ssh-rsa
[15:57:40.955] INFO kex: server: chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc
[15:57:40.955] INFO kex: server: chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc
[15:57:40.955] INFO kex: server: hmac-sha2-256,hmac-sha2-512,hmac-sha1
[15:57:40.955] INFO kex: server: hmac-sha2-256,hmac-sha2-512,hmac-sha1
[15:57:40.955] INFO kex: server: none,zlib,zlib@openssh.com
[15:57:40.955] INFO kex: server: none,zlib,zlib@openssh.com
[15:57:40.955] INFO kex: server: 
[15:57:40.955] INFO kex: server: 
[15:57:40.955] INFO kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
[15:57:40.955] INFO kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
[15:57:40.955] INFO kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
[15:57:40.955] INFO kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
[15:57:40.955] INFO kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[15:57:40.955] INFO kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[15:57:40.955] INFO kex: client: none
[15:57:40.955] INFO kex: client: none
[15:57:40.955] INFO kex: client: 
[15:57:40.955] INFO kex: client: 
[15:57:40.955] INFO kex: server->client aes128-ctr hmac-sha1 none
[15:57:40.955] INFO kex: client->server aes128-ctr hmac-sha1 none
[15:57:40.957] INFO SSH_MSG_KEX_ECDH_INIT sent
[15:57:40.957] INFO expecting SSH_MSG_KEX_ECDH_REPLY
[15:57:41.070] INFO ssh_rsa_verify: signature true
[15:57:41.079] WARN Permanently added 'github.com' (RSA) to the list of known hosts.
[15:57:41.080] INFO SSH_MSG_NEWKEYS sent
[15:57:41.080] INFO SSH_MSG_NEWKEYS received
[15:57:41.087] INFO SSH_MSG_SERVICE_REQUEST sent
[15:57:41.182] INFO SSH_MSG_SERVICE_ACCEPT received
[15:57:41.278] INFO Authentications that can continue: publickey,keyboard-interactive,password
[15:57:41.278] INFO Next authentication method: publickey
[15:57:41.279] INFO Disconnecting from github.com port 22
Auth fail
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.commandFailed(CommandUtil.java:71)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.checkCommandFailed(CommandUtil.java:38)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.runCommand(CommandUtil.java:117)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.GitCommandLine.run(GitCommandLine.java:131)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.FetchCommandImpl.call(FetchCommandImpl.java:111)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterImpl.fetch(UpdaterImpl.java:693)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.fetchMirror(UpdaterWithMirror.java:156)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.updateLocalMirror(UpdaterWithMirror.java:136)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.updateLocalMirror(UpdaterWithMirror.java:71)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.doUpdate(UpdaterWithMirror.java:63)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterImpl.update(UpdaterImpl.java:129)
    at jetbrains.buildServer.buildTriggers.vcs.git.agent.GitAgentVcsSupport.updateSources(GitAgentVcsSupport.java:110)
    at jetbrains.buildServer.agent.impl.vcs.AgentVcsManagerExImpl$CheckoutSupportImpl.updateSources(AgentVcsManagerExImpl.java:108)
    at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent$1.run(ProjectSourcesOnAgent.java:186)
    at java.lang.Thread.run(Thread.java:748)
[2018-07-29 15:57:41,505]   WARN -    jetbrains.buildServer.AGENT - Failed to obtain artifacts publishing limits. Response from the server: Auth failed
[2018-07-29 15:57:41,505]   WARN - ernalArtifactsProcessingLogger - Failed to publish artifacts: jetbrains.buildServer.agent.ArtifactPublishingFailedException: Auth failed (enable debug to see stacktrace)

As there is no problem and all going well when using the default build agent, I assume it is related to your plugin. Do you have any idea / hint, what could cause this?

JeanRev commented 6 years ago

Hi @Vampire :-) Looks like the Teamcity built-in git client cannot access your SSH key-pair for authentication on github. Did you specify an absolute path to your SSH key and certificates in your VCS-Root settings? Or is Teamcity configured to use the "Default" private key of your system? If yes, this might explains why you are seeing this. If you can, I would try uploading the SSH key files to your Teamcity instance instead, this should definitively work (the TC will transmit it automatically to the build agent). Otherwise, you will need to bind the key files somehow in the container (they will need to be available on the server running the Docker daemon).

The other warnings are maybe related, I would first ignore them.

Vampire commented 6 years ago

Neither, nor. I'm already using uploaded key. You see the second line, where it tries to get it, but is denied with 403 forbidden.

JeanRev commented 6 years ago

Interesting. The warning about the missing ssl certificate is puzzling. Did you set up certificate based authentication between the client and server?

I would try to start the container manually to see if you can reproduce the issue (instructions for the official agent image are available here: https://hub.docker.com/r/jetbrains/teamcity-agent/ ) Note that your agent will register itself as a "regular" agent and you'll need to authorize it explicitly.

I doubt that the problem comes from the plugin though, it can't really disrupt the communication between the agent and server.

Vampire commented 6 years ago

Ok, forget it. It is kind of a homemade problem. We have a proxy in front of TeamCity that teminates https and also adds guest=1 parameter to all requests, so that 99% of the users of this instance do not always have to press the "Login as guest" link which will scare away many users before they even press it.

For normal logged in users this is no problem, as there the parameter then is ignored and they can work in their session.

But the agents did not like this when doing stuff that needs priviledges like requesting SSH keys.

For now I changed the agents to run on "Host" network and access TC on 127.0.0.1, this way it works properly.

Maybe in the future I also move TC into a Docker container and then have them share a network in which they can talk.