How to spin up agents?

[tumbl3w33d]

Summary

Your documentation on how to configure the cloud in Jenkins is fine, but I couldn't find any hint how to actually spin up agents. From the k8s plugin I know that I can use a dedicated agent kind in declarative pipelines but there is none for hetzner. So, a snippet how to actually use hcloud slaves from a declarative pipeline would be great.

[tumbl3w33d]

I just realized that the 'cloud statistics' section in the Jenkins UI shows that the plugin keeps trying to spin up an agent, but it's not clear to me what triggers these attempts because I do nothing special in my example pipeline what I would expect to cause this.

Unrelated - these attempts fail with an exception:

java.lang.IllegalStateException: Invalid API response : 409
    at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
    at cloud.dnation.jenkins.plugins.hetzner.Helper.assertValidResponse(Helper.java:117)
    at cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager.getOrCreateSshKey(HetznerCloudResourceManager.java:170)
    at cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager.createServer(HetznerCloudResourceManager.java:200)
    at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:42)
    at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:30)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

This helped me to figure out I shouldn't have created a SSH key in the hcloud UI manually before. After I deleted that, the jenkins plugin created it automatically and started spinning up the agent. I just still don't understand why.

The next provisioning failure was

java.lang.IllegalStateException: Computer is not connected : hcloud-yw4vjvevsclcodb2
    at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:77)
    at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:30)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

and I didn't figure out that one yet. Might have something to do with the hcloud base image that I configured.

[tumbl3w33d]

Alright, I guess I understood that the label you define in the server template is what creates a listener and when you use that label in your pipeline (as agent label) then the agent gets created on-demand.

The next issue I faced is

ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
java.io.IOException: Remote scp terminated with error (scp: /home/jenkins: No such file or directory).
    at com.trilead.ssh2.SCPClient.readResponse(SCPClient.java:51)
    at com.trilead.ssh2.SCPClient.sendBytes(SCPClient.java:135)
    at com.trilead.ssh2.SCPClient.put(SCPClient.java:514)
Also:   java.lang.Throwable: launched here
    at hudson.slaves.SlaveComputer._connect(SlaveComputer.java:282)
    at hudson.model.Computer.connect(Computer.java:440)
    at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:67)
    at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:30)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException: Error during SCP transfer.
    at com.trilead.ssh2.SCPClient.put(SCPClient.java:518)
    at cloud.dnation.jenkins.plugins.hetzner.launcher.HetznerServerComputerLauncher.copyAgent(HetznerServerComputerLauncher.java:83)
    at cloud.dnation.jenkins.plugins.hetzner.launcher.HetznerServerComputerLauncher.launch(HetznerServerComputerLauncher.java:102)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:293)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

I didn't expected that a directory /home/jenkins need to exist on the target VM. Since the plugin is connecting as root, I expected the provisioning would do whatever necessary to get the agent running. But I guess it has to do with what is configured in the server template - Connect via SSH as root, but launch agent as user configured in credentials so it's related to the user that is configured in the server template's configured ssh credentials and I would probably need to create some with user root as next step.

[rkosegi]

Hi, thank you for this report.

I couldn't find any hint how to actually spin up agents.

Will update documentation that makes it clear

I didn't expected that a directory /home/jenkins need to exist on the target VM

That's actually required. In theory this directory could be created by Plugin, but only in situation when connection is made by root, which will not solve problem when connecting as regular user. Having such special case will only lead to more confusion.

I recommend to use Hashicorp packer to create image with everything required, we have an example.