Agents fail to start with `Permission denied` error while executing SCP command

Jenkins and plugins versions report

Environment

```text Jenkins: 2.419 OS: Linux - 6.3.12-200.fc38.x86_64 Java: 17.0.8 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ansicolor:1.0.3 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 apache-httpcomponents-client-5-api:5.2.1-1.0 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 basic-branch-build-strategies:81.v05e333931c7d bootstrap5-api:5.3.0-1 bouncycastle-api:2.29 branch-api:2.1122.v09cb_8ea_8a_724 build-timeout:1.31 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.0 cloud-stats:316.vd6d6b_292238d cloudbees-folder:6.848.ve3b_fd7839a_81 code-coverage-api:4.7.0 command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.10.0-68.v0d0b_c439292b_ config-file-provider:953.v0432a_802e4d2 copyartifact:714.v28a_34f8c563f credentials:1271.v54b_1c2c6388a_ credentials-binding:631.v861c06d062b_4 dark-theme:359.vb_d6175e5f6f9 data-tables-api:1.13.5-1 display-url-api:2.3.9 docker-commons:439.va_3cb_0a_6a_fb_29 docker-java-api:3.3.1-79.v20b_53427e041 docker-plugin:1.4 docker-workflow:572.v950f58993843 dockerhub-notification:2.7.0 durable-task:523.va_a_22cf15d5e0 echarts-api:5.4.0-5 embeddable-build-status:412.v09da_db_1dee68 font-awesome-api:6.4.0-2 forensics-api:2.3.0 git:5.2.0 git-client:4.4.0 github:1.37.3 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1732.v3f1889a_c475b_ github-checks:545.v79a_a_68b_ca_682 google-compute-engine:4.3.16 google-oauth-plugin:1.0.9 hetzner-cloud:999999-SNAPSHOT (private-e3d68a85-mruder) instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 jjwt-api:0.11.5-77.v646c772fddb_0 jquery3-api:3.7.0-1 jsch:0.2.8-65.v052c39de79b_2 junit:1217.v4297208a_a_b_ce lockable-resources:1185.v0c528656ce04 mailer:463.vedf8358e006b_ matrix-project:808.v5a_b_5f56d6966 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ monitoring:1.95.0 nodejs:1.6.1 oauth-credentials:0.645.ve666a_c332668 okhttp-api:4.11.0-157.v6852a_a_fa_ec11 pipeline-build-step:505.v5f0844d8d126 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:671.v07c339c842e8 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 pipeline-stage-view:2.33 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.3.0 prism-api:1.29.0-7 resource-disposer:0.23 role-strategy:680.v3a_6a_1698b_864 scm-api:676.v886669a_199a_a_ script-security:1269.v639888f5e366 simple-theme-plugin:160.vb_76454b_67900 snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 ssh-agent:333.v878b_53c89511 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.916.vd17b_43357ce4 ssh-steps:2.0.65.vd26b_5b_9b_de4d sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 theme-manager:209.va_da_1152274b_e timestamper:1.26 token-macro:384.vf35b_f26814ec trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1259.vb_47f14fffc8a_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3769.v8b_e595e4d40d workflow-durable-task-step:1284.v4fcd365b_75b_e workflow-job:1339.v7ede8a_29ed06 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:848.v5a_383b_d14921 ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Docker container jenkins/jenkins:alpine-jdk17 based on Alpine Linux 3.18.3 (Docker host running on Fedora CoreOS 38.20230722.3.0) Agent: Hetzner Cloud image docker-ce based on Ubuntu 22.04

Reproduction steps

Create a server template with
- image docker-ce
- server type cpx41
- location fsn1
and the following cloud-init snippet:
```
#cloud-config
[…]
packages:
- default-jre-headless
package_update: true
```

When trying to run a job on that agent, I suspect that the following happens:

The server is provisioned and starts
Jenkins connects to the server via SSH and attempts to bring the agent online
1. Jenkins copies the agent JAR to the server and sets its permissions to 0444 (read-only for everyone)
2. Jenkins copies the agent launch script to the server and sets its permissions to 0555 (read-only and executable for everyone)
3. Jenkins attempts to start the agent by running the command java -jar …/remoting.jar, which fails because the JRE is not yet installed on the server
cloud-init finishes installing the JRE on the server
After some seconds, Jenkins connects to the server via SSH and attempts to bring the agent online again

Expected Results

Jenkins is able to copy the agent JAR and launch script without a problem and brings the agent online.

Actual Results

Jenkins fails to copy the agent JAR to the server and prints a Permission denied error. This is because the agent JAR has been copied to the server before and exists with read-only permissions so that the second attempt to copy (and overwrite) it fails.

Anything else?

This error occurs in about 10–20 percent of the times an instance is provisioned. Sometimes installing the JRE is finished before Jenkins connects to the server, sometimes not. Yes, I could create a custom image with Java pre-installed, but installing it using cloud-init on a Hetzner off-the-shelf image is faster (10–15 seconds) than Hetzner needs to copy the custom image onto the server (1 minute).

jenkinsci / hetzner-cloud-plugin