search

jenkinsci / openstack-cloud-plugin

Provision nodes from OpenStack on demand
https://plugins.jenkins.io/openstack-cloud
MIT License
47 stars 83 forks source link

Test connection fails with connect timed out while fetching network #320

Closed tyge68 closed 3 years ago

tyge68 commented 3 years ago

Version report

Jenkins and plugins versions report:

Jenkins: 2.277.1
OS: Linux - 3.10.0-1160.15.2.el7.x86_64
---
lockable-resources:2.10
build-blocker-plugin:1.7.7
favorite:2.3.3
jquery3-api:3.5.1-3
pipeline-input-step:2.12
http_request:1.8.27
cloudbees-disk-usage-simple:0.10
blueocean-rest-impl:1.24.4
branch-api:2.6.3
icon-shim:3.0.0
antisamy-markup-formatter:2.1
build-timeout:1.20
credentials-binding:1.24
jquery:1.12.4-1
blueocean-core-js:1.24.4
github:1.33.1
javadoc:1.6
hashicorp-vault-plugin:3.7.0
durable-task:1.35
badge:1.8
blueocean-config:1.24.4
structs:1.22
command-launcher:1.5
build-name-setter:2.1.0
blueocean-events:1.24.4
slack:2.46
run-condition:1.5
scm-api:2.6.4
pipeline-stage-tags-metadata:1.8.4
swarm:3.25
instant-messaging:1.42
maven-plugin:3.10
ec2:1.56
gradle:1.36
workflow-aggregator:2.6
hashicorp-vault-pipeline:1.3
blueocean-display-url:2.4.1
simple-theme-plugin:0.6
pipeline-model-definition:1.8.4
aws-credentials:1.28
blueocean:1.24.4
pipeline-stage-step:2.5
pipeline-graph-analysis:1.10
workflow-support:3.8
jobConfigHistory:2.26
junit:1.49
matrix-project:1.18
pam-auth:1.6
openstack-cloud:2.57
git-parameter:0.9.13
docker-commons:1.17
promoted-builds:3.9
bootstrap4-api:4.6.0-2
token-macro:2.15
workflow-scm-step:2.12
ghprb:1.42.2
display-url-api:2.3.4
pipeline-model-extensions:1.8.4
workflow-durable-task-step:2.38
cloudfoundry:2.3.2
cloudbees-bitbucket-branch-source:2.9.7
node-iterator-api:1.5.0
blueocean-autofavorite:1.2.4
violation-comments-to-github:1.94
config-file-provider:3.7.0
email-ext:2.82
mask-passwords:3.0
windows-slaves:1.7
jira:3.2
extended-read-permission:3.2
slave-status:1.6
rebuild:1.32
conjur-credentials:1.0.2
jdk-tool:1.5
gitea:1.2.1
nodejs:1.3.11
mapdb-api:1.0.9.0
plain-credentials:1.7
pipeline-githubnotify-step:1.0.5
credentials:2.3.15
resource-disposer:0.15
cloud-stats:0.26
git-client:3.6.0
clover:4.11.1
kubernetes-client-api:4.13.2-1
blueocean-pipeline-api-impl:1.24.4
jackson2-api:2.12.1
apache-httpcomponents-client-4-api:4.5.13-1.0
handlebars:1.1.1
audit-trail:3.8
pubsub-light:1.13
conditional-buildstep:1.4.1
ivy:2.1
pipeline-github:2.7
blueocean-i18n:1.24.4
workflow-multibranch:2.22
variant:1.4
mercurial:2.12
pipeline-model-declarative-agent:1.1.1
snakeyaml-api:1.27.0
blueocean-rest:1.24.4
aws-java-sdk:1.11.955
artifactory:3.10.5
script-security:1.76
pipeline-stage-view:2.19
workflow-job:2.40
configuration-as-code:1.47
jaxb:2.3.0.1
git-server:1.9
workflow-cps:2.90
font-awesome-api:5.15.2-2
pipeline-utility-steps:2.6.1
blueocean-pipeline-editor:1.24.4
localization-zh-cn:1.0.24
cloudbees-folder:6.15
envinject-api:1.7
ssh-agent:1.21
pipeline-milestone-step:1.3.2
localization-support:1.1
job-dsl:1.77
cmakebuilder:2.6.3
blueocean-commons:1.24.4
saml:2.0.0
prometheus:2.0.8
kubernetes:1.29.2
plugin-util-api:2.0.0
workflow-basic-steps:2.23
basic-branch-build-strategies:1.3.2
sonar:2.13
sse-gateway:1.24
build-user-vars-plugin:1.7
blueocean-personalization:1.24.4
docker-plugin:1.2.2
blueocean-git-pipeline:1.24.4
copyartifact:1.46
blueocean-jwt:1.24.4
echarts-api:5.0.1-1
ldap:2.4
okhttp-api:3.14.9
plot:2.1.9
bouncycastle-api:2.20
parameterized-trigger:2.40
handy-uri-templates-2-api:2.1.8-1.0
envinject:2.4.0
job-import-plugin:3.4
pipeline-build-step:2.13
blueocean-github-pipeline:1.24.4
docker-workflow:1.26
popper-api:1.16.1-2
momentjs:1.1.1
metrics:4.0.2.7
ssh-credentials:1.18.1
workflow-step-api:2.23
jenkins-design-language:1.24.4
github-branch-source:2.10.2
ssh-slaves:1.31.5
htmlpublisher:1.25
workflow-api:2.41
pipeline-model-api:1.8.4
ws-cleanup:0.39
p4:1.11.3
github-pullrequest:0.2.8
git:4.6.0
jsch:0.1.55.2
jjwt-api:0.11.2-9.c8b45b8bb173
timestamper:1.11.8
checks-api:1.6.0
authentication-tokens:1.4
workflow-cps-global-lib:2.18
blueocean-web:1.24.4
cvs:2.18
matrix-auth:2.6.5
docker-java-api:3.1.5.2
blueocean-pipeline-scm-api:1.24.4
kubernetes-credentials:0.8.0
jquery-detached:1.2.1
active-directory:2.23
monitoring:1.86.0
ant:1.11
ace-editor:1.1
conjur-simple-integration:0.0.1
greenballs:1.15.1
trilead-api:1.0.13
blueocean-bitbucket-pipeline:1.24.4
role-strategy:3.1
translation:1.16
jquery-ui:1.0.2
parameter-separator:1.3
blueocean-jira:1.24.4
github-pr-comment-build:2.3
pipeline-rest-api:2.19
subversion:2.14.0
mailer:1.33
blueocean-dashboard:1.24.4
jx-pipelines:1.0.15
external-monitor-job:1.7
github-api:1.123
nodelabelparameter:1.7.2
m2release:0.16.2
ansicolor:0.7.5
disk-usage:0.28
groovy-postbuild:2.5
sidebar-link:1.7
groovy:2.3
Paste here

Reproduction steps

Results

Expected result:

No error should occurs, it should be validating the connection.

Actual result:

While configuring the plugin for the first time, it fails while collecting network list apparently with timeout.

Connection not validated, plugin might not operate correctly: connect timed out
java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at okhttp3.internal.platform.Platform.connectSocket(Platform.java:125)
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:238)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:158)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
    at okhttp3.RealCall.execute(RealCall.java:77)
    at org.openstack4j.connectors.okhttp.HttpCommand.execute(HttpCommand.java:130)
    at org.openstack4j.connectors.okhttp.HttpExecutorServiceImpl.invokeRequest(HttpExecutorServiceImpl.java:60)
    at org.openstack4j.connectors.okhttp.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:53)
Caused: ConnectionException{message=connect timed out, status=0, request=GET https://or1sz2-openstack.myhost.com:9696/v2.0/networks}
    at org.openstack4j.connectors.okhttp.HttpExecutorServiceImpl.invoke(HttpExecutorServiceImpl.java:55)
    at org.openstack4j.connectors.okhttp.HttpExecutorServiceImpl.execute(HttpExecutorServiceImpl.java:30)
    at org.openstack4j.core.transport.internal.HttpExecutor.execute(HttpExecutor.java:54)
    at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:211)
    at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:205)
    at org.openstack4j.openstack.networking.internal.NetworkServiceImpl.list(NetworkServiceImpl.java:50)
    at jenkins.plugins.openstack.compute.internal.Openstack.sanityCheck(Openstack.java:761)
    at jenkins.plugins.openstack.compute.JCloudsCloud$DescriptorImpl.doTestConnection(JCloudsCloud.java:556)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
    at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
    at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

Using openstack cli, it works perfectly fine the command 'openstack network list' for example. It's not clear why it fails here. Any pointer what else I could try or if this is definitely a bug ?

olivergondza commented 3 years ago

The neutron endpoint[1] have not responded within 20 seconds, which is usually a symptom of a serious problem on an OpenStack side.

Can you reproduce this repeatedly? How long does it take for the console command to list the networks? How many networks does it list?

[1] https://github.com/jenkinsci/openstack-cloud-plugin/blob/parent-2.57/plugin/src/main/java/jenkins/plugins/openstack/compute/internal/Openstack.java#L761

tyge68 commented 3 years ago

@olivergondza on openstack cli it's instant to list the network, and it returns 2 networks.

So maybe it's more likely a network issue in our infrastructure , where it could login but then it couldn't access openstack network api port from the machine that run jenkins primary. I will check that with our teams.

tyge68 commented 3 years ago

@olivergondza sorry for the noise, it was clearly due to network issue on our infrastructure, I moved the jenkins primary directly inside openstack and there the connection is working.