search

jenkinsci / digitalocean-plugin

Jenkins DigitalOcean Agent Plugin
https://plugins.jenkins.io/digitalocean-plugin/
MIT License
46 stars 38 forks source link

DigitalOceanComputerLauncher: [ED25519] PEM problem: it is of unknown type #65

Closed speedythesnail closed 2 years ago

speedythesnail commented 2 years ago

Jenkins and plugins versions report

Environment Jenkins: 2.361.2 OS: Linux - 5.15.0-48-generic --- ace-editor:1.1 analysis-model-api:10.17.0 ant:481.v7b_09e538fcca antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-timeout:1.24 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.5 cloud-stats:0.27 cloudbees-folder:6.758.vfd75d09eea_a_1 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.9-19.v8df45c678366 conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1512.vb_79d418d5fc8 credentials:1189.vf61b_a_5e2f62e credentials-binding:523.vd859a_4b_122e6 dashboard-view:2.447.vda_124dd35f11 data-tables-api:1.12.1-4 digitalocean-plugin:1.3.1 display-url-api:2.3.6 durable-task:501.ve5d4fc08b0be echarts-api:5.4.0-1 email-ext:2.91 embeddable-build-status:255.va_d2370ee8fde extended-read-permission:3.2 font-awesome-api:6.2.0-3 forensics-api:1.16.0 git:4.12.1 git-client:3.12.1 git-parameter:0.9.18 github:1.35.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1695.v88de84e9f6b_9 gradle:1.40 htmlpublisher:1.31 instance-identity:116.vf8f487400980 ionicons-api:28.va_f3a_84439e5f jackson2-api:2.13.3-285.vc03c0256d517 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.6-2 jdk-tool:55.v1b_32b_6ca_f9ca jjwt-api:0.11.5-77.v646c772fddb_0 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1150.v5c2848328b_60 ldap:2.12 mailer:438.v02c7f0a_12fa_4 matrix-auth:3.1.5 matrix-project:785.v06b_7f47b_c631 mina-sshd-api-common:2.9.1-44.v476733c11f82 mina-sshd-api-core:2.9.1-44.v476733c11f82 momentjs:1.1.1 nodejs:1.5.1 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 parameterized-trigger:2.45 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:612.v84da_9c54906d pipeline-input-step:451.vf1a_a_4f405289 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-rest-api:2.25 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.25 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.18.0 popper2-api:2.11.6-2 prism-api:1.29.0-1 rebuild:1.34 resource-disposer:0.20 run-condition:1.5 scm-api:621.vda_a_b_055e58f7 script-security:1183.v774b_0b_0a_a_451 snakeyaml-api:1.32-86.ve3f030a_75631 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:305.v8f4381501156 ssh-slaves:2.846.v1b_70190624f5 sshd:3.249.v2dc2ea_416e33 structs:324.va_f5d6774f3a_d timestamper:1.20 token-macro:308.v4f2b_ed62b_b_16 trilead-api:2.72.v2a_3236754f73 variant:59.vf075fe829ccb warnings-ng:9.20.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1192.v2d0deb_19d212 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:2802.v5ea_628154b_c2 workflow-durable-task-step:1199.v02b_9244f8064 workflow-job:1239.v71b_b_a_124a_725 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:838.va_3a_087b_4055b ws-cleanup:0.43

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu 22.04 x64

Reproduction steps

Expected Results

Jenkins spins up a usable agent using an ED25519 SSH Key generated with ssh-keygen -t ed25519.

Actual Results

Jenkins gets stuck in an endless loop of creating and destroying the droplet, as it is unable to connect. The following exception message occurs:

Oct 10, 2022 4:20:43 PM WARNING com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher launch

Publickey authentication failed.
java.io.IOException: PEM problem: it is of unknown type. Supported algorithms are :[ssh-ed25519, ecdsa-sha2-nistp521, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss]
    at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:482)
    at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:290)
Caused: java.io.IOException: Publickey authentication failed.
    at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:349)
    at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
    at com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher.launch(DigitalOceanComputerLauncher.java:170)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Anything else?

I am able to connect to the newly created droplet by SSH'ing from the Jenkins master node to the agent, using the SSH key I have saved in Jenkins.

I currently have the following configuration for the cloud agent: Instance cap: 2 Timeout in minutes: 10 Connection retry wait in seconds: 30 Image: Ubuntu 22.04 x64 Run as user: root Jenkins workspace directory path: /jenkins/ SSH port: 22 Labels [none] Setup Private Networking: Yes Allow jobs with no label restriction: Yes Number of executors: 2 Idle termination time: -1 Instance cap: 2 Install monitoring: Yes

User data:

#cloud-config
apt_upgrade: true
package_upgrade: true
packages:
  - openjdk-11-jre-headless
  - docker.io
runcmd:
  - [ mkdir, /jenkins ]
  - [ chmod, 777, /jenkins ]

Init script:

#!/bin/bash

echo "starting init script"
while ! cloud-init status|grep -qF 'done'
do
  echo "waiting for cloud-init to be done"
  sleep 10
done
halkeye commented 2 years ago

Ubuntu 22.04 x64

The default ubuntu images now don't allow certain ssh key types (via /etc/ssh/sshd_config). I eventually regenerated my key with the type of ed25519

Its not really a jenkins bug though.

speedythesnail commented 2 years ago

I'm going to try changing the SSH key type to see if t his solves the issue.

I added a logger for the com.dubture.jenkins.digitalocean package to capture all logs, here's the output of it before I killed the build:

Oct 10, 2022 4:37:36 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate 

Creating SlaveTemplate with imageId = ubuntu-22-04-x64, sizeId = s-1vcpu-2gb, regionId = nyc1

Oct 10, 2022 4:37:36 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud 

Constructing new DigitalOceanCloud(name = DigitalOcean, <token>, <privateKey>, <keyId>, instanceCap = 2, ...)

Oct 10, 2022 4:37:36 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud 

Creating DigitalOcean cloud with 1 templates

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud canProvision

canProvision null :: true

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets

Listing all droplets

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal

cloud limit check

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote

cloud limit check

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal

agent limit check

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote

agent limit check

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud provision

Provisioning 1 DigitalOcean nodes

Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets

Listing all droplets

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal

cloud limit check

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote

cloud limit check

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision

Provisioning agent...

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision

Starting to provision digital ocean droplet using image: ubuntu-22-04-x64, sizeId = s-1vcpu-2gb, regionId = nyc1

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal

agent limit check

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote

agent limit check

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision

Creating agent with new droplet jenkins-DigitalOcean-jenkins.slave-ee73f52f-5a8e-4460-a37b-b2461b3afb73

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate newSlave

Creating new agent...

Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet

Fetching droplet 320368885

Oct 10, 2022 4:54:30 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet

Fetching droplet 320368885

Oct 10, 2022 4:54:31 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress

network 157.230.211.60 => public

Oct 10, 2022 4:54:31 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress

network 10.116.0.3 => private

Oct 10, 2022 4:55:03 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet

Fetching droplet 320368885

Oct 10, 2022 4:55:03 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress

network 157.230.211.60 => public

Oct 10, 2022 4:55:03 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress

network 10.116.0.3 => private

Oct 10, 2022 4:55:05 PM WARNING com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher launch

Publickey authentication failed.
java.io.IOException: PEM problem: it is of unknown type. Supported algorithms are :[ssh-ed25519, ecdsa-sha2-nistp521, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss]
    at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:482)
    at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:290)
Caused: java.io.IOException: Publickey authentication failed.
    at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:349)
    at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
    at com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher.launch(DigitalOceanComputerLauncher.java:170)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputer onRemoved

Agent removed, deleting droplet 320368885

Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean tryDestroyDropletAsync

Adding droplet to destroy 320368885

Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean lambda$static$0

Trying to destroy droplet 320368885

Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud canProvision

canProvision jenkins-DigitalOcean-jenkins.slave-ee73f52f-5a8e-4460-a37b-b2461b3afb73 :: false

Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean lambda$static$0

Droplet 320368885 is destroyed

Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean lambda$static$0

Waiting on more droplets to destroy

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud canProvision

canProvision null :: true

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets

Listing all droplets

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal

cloud limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote

cloud limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal

agent limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote

agent limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud provision

Provisioning 1 DigitalOcean nodes

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets

Listing all droplets

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal

cloud limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote

cloud limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision

Provisioning agent...

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision

Starting to provision digital ocean droplet using image: ubuntu-22-04-x64, sizeId = s-1vcpu-2gb, regionId = nyc1

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal

agent limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote

agent limit check

Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision

Creating agent with new droplet jenkins-DigitalOcean-jenkins.slave-653aee96-b069-4eab-bf90-81d426f6b6e2

Oct 10, 2022 4:55:09 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate newSlave

Creating new agent...

Oct 10, 2022 4:55:09 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet

Fetching droplet 320369016
speedythesnail commented 2 years ago

Ubuntu 22.04 x64

The default ubuntu images now don't allow certain ssh key types (via /etc/ssh/sshd_config). I eventually regenerated my key with the type of ed25519

Its not really a jenkins bug though. You're correct, though I don't know if I would not call it a bug, but it would be something worth documenting somewhere in regards to ED25519 keys.

I removed the passphrase from the key and now it works, as described in the below bug report I just came across: JENKINS-46754: 2.73 SSH agent sometimes will not start if using passphrase-protected ed25519 key

The Jenkins 2.73.1 LTS release fails to connect my ssh agents which use an ed25519 passphrase protected private key.  These agents connected successfully with Jenkins 2.60.3 LTS and earlier.

I've confirmed that dsa passphrase protected private keys work in all cases and that rsa passphrase protected private keys work in all cases. The rsa private keys and ed25519 private keys which are not passphrase protected work in all cases.

It appears to only be ed25519 private keys which are passphrase protected that have a problem in two of my six tested configurations with 2.73.1 LTS.  Those same configurations work as expected with 2.60.3 LTS.

Thanks for the quick response and I hope I didn't waste you anyone's time!