Closed andycockers closed 2 years ago
rkosegi
commented
2 years ago Hi. This (obscure) error means, that primary IP is chosen based on label selector, however given selector doesn't yields any usable IP address. Usable address must be free (unallocated) and must be within same datacenter location. You can use this link to configure detailed logging where you can check response from Hetzner cloud.
sandrinr
commented
1 year ago I seem to experience the same issue and am puzzled whats going on.
Background: we want to move our Jenkins instance to a new server. On the new server the Hetzner plugin does not want to function with the error printed in the leading post of this issue. On the existing instance, with seemingly the same config for the plugin it works fine.
When I look into the detailed Hetzner plugin log I see:
Creating new agent with 1 executors, have 0 running VMs
--> GET https://api.hetzner.cloud/v1/ssh_keys?label_selector=jenkins.io%2Fmanaged-by%3Dhetzner-jenkins-plugin%2Cjenkins.io%2Fcredentials-id%3D<redacted>
<-- 200 https://api.hetzner.cloud/v1/ssh_keys?label_selector=jenkins.io%2Fmanaged-by%3Dhetzner-jenkins-plugin%2Cjenkins.io%2Fcredentials-id%3D<redacted>
[...]
{
"ssh_keys": [
{
"id": 7980871,
"name": "3e575aa1-576c-4f95-8526-5c5061c376bc",
"fingerprint": "<redacted>",
"public_key": "<redacted>"
"labels": {
"jenkins.io/managed-by": "hetzner-jenkins-plugin",
"jenkins.io/credentials-id": "3e575aa1-576c-4f95-8526-5c5061c376bc"
},
"created": "2022-09-15T07:29:25+00:00"
}
],
"meta": {
"pagination": {
"page": 1,
"per_page": 25,
"previous_page": null,
"next_page": null,
"last_page": 1,
"total_entries": 1
}
}
}
--> GET https://api.hetzner.cloud/v1/primary_ips?label_selector=
<-- 200 https://api.hetzner.cloud/v1/primary_ips?label_selector=
{
"primary_ips": [],
"meta": {
"pagination": {
"page": 1,
"per_page": 25,
"previous_page": null,
"next_page": null,
"last_page": 1,
"total_entries": 0
}
}
}
--> GET https://api.hetzner.cloud/v1/servers?label_selector=jenkins.io%2Fmanaged-by%3Dhetzner-jenkins-plugin%2Cjenkins.io%2Fcloud-name%3Dhetzner-cloud-cicd&page=1&per_page=50
<-- 200 https://api.hetzner.cloud/v1/servers?label_selector=jenkins.io%2Fmanaged-by%3Dhetzner-jenkins-plugin%2Cjenkins.io%2Fcloud-name%3Dhetzner-cloud-cicd&page=1&per_page=50
{
"servers": [],
"meta": {
"pagination": {
"page": 1,
"per_page": 50,
"previous_page": null,
"next_page": null,
"last_page": 1,
"total_entries": 0
}
}
}It seems the plugin does not attempt to create a server and then trips over the fact that there are no public IPs available.
The standard Jenkins log shows:
2022-09-15 07:51:04.694+0000 [id=58] INFO c.d.j.p.hetzner.HetznerCloud#provision: Creating new agent with 1 executors, have 0 running VMs
2022-09-15 07:51:04.696+0000 [id=58] INFO h.s.NodeProvisioner$StandardStrategyImpl#apply: Started provisioning hcloud-bz5szu67sxgugy4k from hetzner-cloud-cicd with 1 executors. Remaining excess workload: 0
2022-09-15 07:51:14.538+0000 [id=57] WARNING hudson.slaves.NodeProvisioner#update: Unexpected exception encountered while provisioning agent hcloud-bz5szu67sxgugy4k
java.util.NoSuchElementException: No value present
at java.base/java.util.Optional.get(Optional.java:148)
at cloud.dnation.jenkins.plugins.hetzner.primaryip.AbstractByLabelSelector.applyInternal(AbstractByLabelSelector.java:39)
at cloud.dnation.jenkins.plugins.hetzner.primaryip.AbstractPrimaryIpStrategy.apply(AbstractPrimaryIpStrategy.java:33)
Caused: java.lang.RuntimeException
at cloud.dnation.jenkins.plugins.hetzner.primaryip.AbstractPrimaryIpStrategy.apply(AbstractPrimaryIpStrategy.java:36)
at cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager.lambda$createServer$1(HetznerCloudResourceManager.java:231)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at cloud.dnation.jenkins.plugins.hetzner.HetznerCloudResourceManager.createServer(HetznerCloudResourceManager.java:231)
at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:42)
at cloud.dnation.jenkins.plugins.hetzner.NodeCallable.call(NodeCallable.java:30)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)Any idea whats going on?
sandrinr
commented
1 year ago I think I found a hint. Looking at the config.xml of the two instances I find the following difference:
Not working instance:
<primaryIp class="cloud.dnation.jenkins.plugins.hetzner.primaryip.ByLabelSelectorFailing">
<failIfError>true</failIfError>
<selector></selector>
</primaryIp>Working instance:
<primaryIp class="cloud.dnation.jenkins.plugins.hetzner.primaryip.DefaultStrategy">
<failIfError>false</failIfError>
</primaryIp>However in the GUI the config looks the same.
sandrinr
commented
1 year ago Another update. It turns out this setting did the trick. It is also visible in the GUI but under the advanced section of a server template. Not setting the "default strategy" as the default setting in new setups seems like a bug to me.
rkosegi
commented
1 year ago Hi @sandrinr, you are correct - good analysis. This might be result of incorrect migration of configuration around the time this feature was introduced.There is no fix possible other then re-create template from scratch. Issue will not repeat after template is modified I believe
rkosegi
commented
1 year ago OK, so if it's fixable from GUI, that's even better. Again, this is bug, but it was introduced by older version of plugin and it won't repeat with current version.
Jenkins and plugins versions report
Environment
```text Jenkins: 2.346.3 OS: Linux - 5.15.0-1016-oracle --- ace-editor:1.1 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 authentication-tokens:1.4 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.0-3 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.5 cloud-stats:0.27 cloudbees-folder:6.740.ve4f4ffa_dea_54 command-launcher:84.v4a_97f2027398 configuration-as-code:1512.vb_79d418d5fc8 credentials:1143.vb_e8b_b_ceee347 credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.6 docker-commons:1.21 docker-workflow:521.v1a_a_dd2073b_2e durable-task:500.v8927d9fd99d8 echarts-api:5.3.3-1 extended-read-permission:3.2 font-awesome-api:6.1.2-1 git:4.11.5 git-client:3.11.2 git-server:1.11 google-login:1.6 handlebars:3.0.8 hetzner-cloud:61.v5cf24cc22f1b icon-shim:3.0.0 jackson2-api:2.13.3-285.vc03c0256d517 jakarta-activation-api:2.0.1-1 jakarta-mail-api:2.0.1-1 javax-activation-api:1.2.0-4 javax-mail-api:1.6.2-7 jaxb:2.3.6-1 jdk-tool:55.v1b_32b_6ca_f9ca jquery3-api:3.6.0-4 jsch:0.1.55.61.va_e9ee26616e7 junit:1119.1121.vc43d0fc45561 kubernetes:3704.va_08f0206b_95e kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 kubernetes-credentials-provider:1.196.va_55f5e31e3c2 lockable-resources:2.16 mailer:435.v79ef3972b_5c7 matrix-project:772.v494f19991984 metrics:4.2.10-389.v93143621b_050 mina-sshd-api-common:2.8.0-36.v8e25ce90d4b_1 mina-sshd-api-core:2.8.0-36.v8e25ce90d4b_1 momentjs:1.1.1 pipeline-build-step:2.18 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:612.v84da_9c54906d pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-rest-api:2.24 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.24 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.6-1 scm-api:621.vda_a_b_055e58f7 script-security:1175.v4b_d517d6db_f0 snakeyaml-api:1.31-84.ve43da_fb_49d0b ssh-credentials:295.vced876c18eb_4 ssh-slaves:1.834.v622da_57f702c sshd:3.242.va_db_9da_b_26a_c3 structs:324.va_f5d6774f3a_d trilead-api:1.67.vc3938a_35172f variant:59.vf075fe829ccb workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1192.v2d0deb_19d212 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:2759.v87459c4eea_ca_ workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1199.v02b_9244f8064 workflow-job:1207.ve6191ff089f8 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:838.va_3a_087b_4055b ```What Operating System are you using (both controller, and any agents involved in the problem)?
Jenkins is running in bare metal Kubernetes on an ARM64 VM in Oracle cloud. The host OS is Ubuntu 20.04. The agents are running Ubuntu 20.04
Reproduction steps
Expected Results
The node would provision, the job would run and the server would be deleted.
Actual Results
This error is thrown in the hetzner cloud stats:
Anything else?
No response