Closed speedythesnail closed 2 years ago
Ubuntu 22.04 x64
The default ubuntu images now don't allow certain ssh key types (via /etc/ssh/sshd_config). I eventually regenerated my key with the type of ed25519
Its not really a jenkins bug though.
I'm going to try changing the SSH key type to see if t his solves the issue.
I added a logger for the com.dubture.jenkins.digitalocean package to capture all logs, here's the output of it before I killed the build:
Oct 10, 2022 4:37:36 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate
Creating SlaveTemplate with imageId = ubuntu-22-04-x64, sizeId = s-1vcpu-2gb, regionId = nyc1
Oct 10, 2022 4:37:36 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud
Constructing new DigitalOceanCloud(name = DigitalOcean, <token>, <privateKey>, <keyId>, instanceCap = 2, ...)
Oct 10, 2022 4:37:36 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud
Creating DigitalOcean cloud with 1 templates
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud canProvision
canProvision null :: true
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets
Listing all droplets
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal
cloud limit check
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote
cloud limit check
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal
agent limit check
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote
agent limit check
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud provision
Provisioning 1 DigitalOcean nodes
Oct 10, 2022 4:53:58 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets
Listing all droplets
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal
cloud limit check
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote
cloud limit check
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision
Provisioning agent...
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision
Starting to provision digital ocean droplet using image: ubuntu-22-04-x64, sizeId = s-1vcpu-2gb, regionId = nyc1
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal
agent limit check
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote
agent limit check
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision
Creating agent with new droplet jenkins-DigitalOcean-jenkins.slave-ee73f52f-5a8e-4460-a37b-b2461b3afb73
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate newSlave
Creating new agent...
Oct 10, 2022 4:53:59 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet
Fetching droplet 320368885
Oct 10, 2022 4:54:30 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet
Fetching droplet 320368885
Oct 10, 2022 4:54:31 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress
network 157.230.211.60 => public
Oct 10, 2022 4:54:31 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress
network 10.116.0.3 => private
Oct 10, 2022 4:55:03 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet
Fetching droplet 320368885
Oct 10, 2022 4:55:03 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress
network 157.230.211.60 => public
Oct 10, 2022 4:55:03 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher getIpAddress
network 10.116.0.3 => private
Oct 10, 2022 4:55:05 PM WARNING com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher launch
Publickey authentication failed.
java.io.IOException: PEM problem: it is of unknown type. Supported algorithms are :[ssh-ed25519, ecdsa-sha2-nistp521, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss]
at com.trilead.ssh2.crypto.PEMDecoder.decodeKeyPair(PEMDecoder.java:482)
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:290)
Caused: java.io.IOException: Publickey authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:349)
at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
at com.dubture.jenkins.digitalocean.DigitalOceanComputerLauncher.launch(DigitalOceanComputerLauncher.java:170)
at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanComputer onRemoved
Agent removed, deleting droplet 320368885
Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean tryDestroyDropletAsync
Adding droplet to destroy 320368885
Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean lambda$static$0
Trying to destroy droplet 320368885
Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud canProvision
canProvision jenkins-DigitalOcean-jenkins.slave-ee73f52f-5a8e-4460-a37b-b2461b3afb73 :: false
Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean lambda$static$0
Droplet 320368885 is destroyed
Oct 10, 2022 4:55:05 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean lambda$static$0
Waiting on more droplets to destroy
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud canProvision
canProvision null :: true
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets
Listing all droplets
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal
cloud limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote
cloud limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal
agent limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote
agent limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud provision
Provisioning 1 DigitalOcean nodes
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplets
Listing all droplets
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedLocal
cloud limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.DigitalOceanCloud isInstanceCapReachedRemote
cloud limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision
Provisioning agent...
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision
Starting to provision digital ocean droplet using image: ubuntu-22-04-x64, sizeId = s-1vcpu-2gb, regionId = nyc1
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedLocal
agent limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate isInstanceCapReachedRemote
agent limit check
Oct 10, 2022 4:55:08 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate provision
Creating agent with new droplet jenkins-DigitalOcean-jenkins.slave-653aee96-b069-4eab-bf90-81d426f6b6e2
Oct 10, 2022 4:55:09 PM INFO com.dubture.jenkins.digitalocean.SlaveTemplate newSlave
Creating new agent...
Oct 10, 2022 4:55:09 PM INFO com.dubture.jenkins.digitalocean.DigitalOcean getDroplet
Fetching droplet 320369016
Ubuntu 22.04 x64
The default ubuntu images now don't allow certain ssh key types (via /etc/ssh/sshd_config). I eventually regenerated my key with the type of
ed25519
Its not really a jenkins bug though. You're correct, though I don't know if I would not call it a bug, but it would be something worth documenting somewhere in regards to ED25519 keys.
I removed the passphrase from the key and now it works, as described in the below bug report I just came across: JENKINS-46754: 2.73 SSH agent sometimes will not start if using passphrase-protected ed25519 key
The Jenkins 2.73.1 LTS release fails to connect my ssh agents which use an ed25519 passphrase protected private key. These agents connected successfully with Jenkins 2.60.3 LTS and earlier.
I've confirmed that dsa passphrase protected private keys work in all cases and that rsa passphrase protected private keys work in all cases. The rsa private keys and ed25519 private keys which are not passphrase protected work in all cases.
It appears to only be ed25519 private keys which are passphrase protected that have a problem in two of my six tested configurations with 2.73.1 LTS. Those same configurations work as expected with 2.60.3 LTS.
Thanks for the quick response and I hope I didn't waste you anyone's time!
Jenkins and plugins versions report
Environment
Jenkins: 2.361.2 OS: Linux - 5.15.0-48-generic --- ace-editor:1.1 analysis-model-api:10.17.0 ant:481.v7b_09e538fcca antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-timeout:1.24 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.5 cloud-stats:0.27 cloudbees-folder:6.758.vfd75d09eea_a_1 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.9-19.v8df45c678366 conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1512.vb_79d418d5fc8 credentials:1189.vf61b_a_5e2f62e credentials-binding:523.vd859a_4b_122e6 dashboard-view:2.447.vda_124dd35f11 data-tables-api:1.12.1-4 digitalocean-plugin:1.3.1 display-url-api:2.3.6 durable-task:501.ve5d4fc08b0be echarts-api:5.4.0-1 email-ext:2.91 embeddable-build-status:255.va_d2370ee8fde extended-read-permission:3.2 font-awesome-api:6.2.0-3 forensics-api:1.16.0 git:4.12.1 git-client:3.12.1 git-parameter:0.9.18 github:1.35.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1695.v88de84e9f6b_9 gradle:1.40 htmlpublisher:1.31 instance-identity:116.vf8f487400980 ionicons-api:28.va_f3a_84439e5f jackson2-api:2.13.3-285.vc03c0256d517 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.6-2 jdk-tool:55.v1b_32b_6ca_f9ca jjwt-api:0.11.5-77.v646c772fddb_0 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1150.v5c2848328b_60 ldap:2.12 mailer:438.v02c7f0a_12fa_4 matrix-auth:3.1.5 matrix-project:785.v06b_7f47b_c631 mina-sshd-api-common:2.9.1-44.v476733c11f82 mina-sshd-api-core:2.9.1-44.v476733c11f82 momentjs:1.1.1 nodejs:1.5.1 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 parameterized-trigger:2.45 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:612.v84da_9c54906d pipeline-input-step:451.vf1a_a_4f405289 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-rest-api:2.25 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.25 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.18.0 popper2-api:2.11.6-2 prism-api:1.29.0-1 rebuild:1.34 resource-disposer:0.20 run-condition:1.5 scm-api:621.vda_a_b_055e58f7 script-security:1183.v774b_0b_0a_a_451 snakeyaml-api:1.32-86.ve3f030a_75631 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:305.v8f4381501156 ssh-slaves:2.846.v1b_70190624f5 sshd:3.249.v2dc2ea_416e33 structs:324.va_f5d6774f3a_d timestamper:1.20 token-macro:308.v4f2b_ed62b_b_16 trilead-api:2.72.v2a_3236754f73 variant:59.vf075fe829ccb warnings-ng:9.20.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1192.v2d0deb_19d212 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:2802.v5ea_628154b_c2 workflow-durable-task-step:1199.v02b_9244f8064 workflow-job:1239.v71b_b_a_124a_725 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:838.va_3a_087b_4055b ws-cleanup:0.43What Operating System are you using (both controller, and any agents involved in the problem)?
Ubuntu 22.04 x64
Reproduction steps
Expected Results
Jenkins spins up a usable agent using an ED25519 SSH Key generated with ssh-keygen -t ed25519.
Actual Results
Jenkins gets stuck in an endless loop of creating and destroying the droplet, as it is unable to connect. The following exception message occurs:
Anything else?
I am able to connect to the newly created droplet by SSH'ing from the Jenkins master node to the agent, using the SSH key I have saved in Jenkins.
I currently have the following configuration for the cloud agent: Instance cap: 2 Timeout in minutes: 10 Connection retry wait in seconds: 30 Image: Ubuntu 22.04 x64 Run as user: root Jenkins workspace directory path: /jenkins/ SSH port: 22 Labels [none] Setup Private Networking: Yes Allow jobs with no label restriction: Yes Number of executors: 2 Idle termination time: -1 Instance cap: 2 Install monitoring: Yes
User data:
Init script: