search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.1k stars 1.11k forks source link

(HTTP 530, error code 4250) Failed to fetch any free public IP address #7109

Closed ccycv closed 1 year ago

ccycv commented 1 year ago
ISSUE TYPE
COMPONENT NAME
VR, Network
CLOUDSTACK VERSION
4.17.1, 4.17.2
CONFIGURATION
Advanced networking with local storage
OS / ENVIRONMENT
Ubuntu 20.04, with VMware 6.7 hypervisors/vCenter
SUMMARY
STEPS TO REPRODUCE

I was not able to reproduce for any other Type "shared" network ( I have multiple with more than 3 guest ranges / network, in this setup and other ). Here I already have 2 ranges in use and there is still 1 free IP from the 2x guest ranges which where already added, and is showing me this error, even if the 3th range is added or not.

image image image

EXPECTED RESULTS
Not getting this false error
ACTUAL RESULTS
2023-01-18 12:54:27,040 ERROR [c.c.n.IpAddressManagerImpl] (qtp1462044018-530:ctx-8923bb18 ctx-e535fc81 ctx-b0eb5df6) (logid:51501a2d) Failed to fetch any free public IP address
2023-01-18 12:54:27,045 ERROR [c.c.a.ApiServer] (qtp1462044018-530:ctx-8923bb18 ctx-e535fc81 ctx-b0eb5df6) (logid:51501a2d) unhandled exception executing api command: [Ljava.lang.String;@6320ffc6
com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any free public IP address
        at com.cloud.network.IpAddressManagerImpl.lambda$assignAndAllocateIpAddressEntry$0(IpAddressManagerImpl.java:379)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.network.IpAddressManagerImpl.assignAndAllocateIpAddressEntry(IpAddressManagerImpl.java:335)
        at com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:959)
        at com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
        at com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
        at com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
        at com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
        at com.cloud.network.IpAddressManagerImpl$11.doInTransactionWithoutResult(IpAddressManagerImpl.java:2195)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.network.IpAddressManagerImpl.allocateDirectIp(IpAddressManagerImpl.java:2175)
        at com.cloud.network.guru.DirectNetworkGuru$1.doInTransactionWithoutResult(DirectNetworkGuru.java:299)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.network.guru.DirectNetworkGuru.allocateDirectIp(DirectNetworkGuru.java:292)
        at com.cloud.network.guru.DirectNetworkGuru.allocate(DirectNetworkGuru.java:265)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
        at com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
        at com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
        at org.apache.cloudstack.engine.orchestration.CloudOrchestrator.createVirtualMachine(CloudOrchestrator.java:255)
        at com.cloud.vm.UserVmManagerImpl$4.doInTransaction(UserVmManagerImpl.java:4413)
        at com.cloud.vm.UserVmManagerImpl$4.doInTransaction(UserVmManagerImpl.java:4295)
        at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
        at com.cloud.vm.UserVmManagerImpl.commitUserVm(UserVmManagerImpl.java:4295)
        at com.cloud.vm.UserVmManagerImpl.commitUserVm(UserVmManagerImpl.java:4518)
        at com.cloud.vm.UserVmManagerImpl.createVirtualMachine(UserVmManagerImpl.java:4155)
        at com.cloud.vm.UserVmManagerImpl.createAdvancedVirtualMachine(UserVmManagerImpl.java:3668)
        at com.cloud.vm.UserVmManagerImpl.createVirtualMachine(UserVmManagerImpl.java:5776)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy180.createVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.create(DeployVMCmd.java:752)
        at com.cloud.api.dispatch.CommandCreationWorker.handle(CommandCreationWorker.java:47)
        at com.cloud.api.dispatch.DispatchChain.dispatch(DispatchChain.java:37)
        at com.cloud.api.ApiDispatcher.dispatchCreateCmd(ApiDispatcher.java:88)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:700)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:96)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
weizhouapache commented 1 year ago

@ccycv can you please check the database if there is an Ip which is not allocated (allocated=NULL) but not in Free state ?

select * from cloud.user_ip_address where state!='Free' and allocated is null;

ccycv commented 1 year ago

@weizhouapache Yes, looks like I have an IP with allocated=NULL but with state=allocated. Looks like after I have changed the state to free ( it was an unused IP). Thank you for the solution!

weizhouapache commented 1 year ago

@ccycv good to know it. closing this issue then