search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
1.98k stars 1.09k forks source link

Instance Expunge Error #7644

Closed Hossein4185 closed 1 year ago

Hossein4185 commented 1 year ago
ISSUE TYPE

Question

CLOUDSTACK VERSION

4.13.1

CONFIGURATION
OS / ENVIRONMENT

Ubuntu 18.0.4

SUMMARY

I added new management to other Management Serevrs and successfuly Create Instance but when I Expunge the instance I get an 'Command failed due to Internal Server' Error and Stuck in Expunging state

and Cloudstack Server log is: 2023-06-17 17:06:11,055 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-15:ctx-7b3021b7 job-510490 ctx-262372db) (logid:9c8bfc6d) Destroying vm VM[User|i-19870-8018-VM] 2023-06-17 17:06:11,055 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-15:ctx-7b3021b7 job-510490 ctx-262372db) (logid:9c8bfc6d) Cleaning up NICS 2023-06-17 17:06:11,076 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-15:ctx-7b3021b7 job-510490) (logid:9c8bfc6d) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.ExpungeVMCmd java.lang.NullPointerException at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:528) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:494) at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:2203) at com.cloud.vm.UserVmManagerImpl.expungeVm(UserVmManagerImpl.java:4960) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy171.expungeVm(Unknown Source) at org.apache.cloudstack.api.command.admin.vm.ExpungeVMCmd.execute(ExpungeVMCmd.java:109) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-06-17 17:06:11,086 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-15:ctx-7b3021b7 job-510490) (logid:9c8bfc6d) Complete async job-510490, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}

Thanks For guide me

STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
kiranchavala commented 1 year ago

@Hossein4185 Could you please send the complete management server for the date 2023-06-17.

You can upload it google drive and provide the link

DaanHoogland commented 1 year ago

the line of the NPE is


        final VirtualMachineProfile profile = new VirtualMachineProfileImpl(vm);

        final HypervisorGuru hvGuru = _hvGuruMgr.getGuru(vm.getHypervisorType());

        s_logger.debug("Cleaning up NICS");
        final List<Command> nicExpungeCommands = hvGuru.finalizeExpungeNics(vm, profile.getNics());

As per you logs the vm is obviously not null it is either the hvGuru or the profile. Can you check in your DB if the vm has a hypervisor type. If so the generation of the profile failed (no idea why yet). If no it could not find the guru type for this vm.

That all said, please check the versions of all your MSses and compare to what the DB says.

4.13 is quite old. Was it originally the apache version? it could be there is a discrepency in the DB frm migrating from ACP to ACS.

Hossein4185 commented 1 year ago

The following log is the full server error:

2023-06-19 10:34:50,252 DEBUG [c.c.a.ApiServlet] (qtp1709366259-11:ctx-601ff485) (logid:30a99e62) ===START=== 192.168.123.19 -- GET command=queryAsyncJobResult&jobId=c02aa83d-dd78-4922-9a0e-8915c4a73d3b&response=json&=1687158100241 2023-06-19 10:34:50,272 DEBUG [c.c.a.ApiServer] (qtp1709366259-11:ctx-601ff485 ctx-d9ca0fd4) (logid:30a99e62) CIDRs from which account 'Acct[a27ca787-edb8-4361-b690-cb2f4cd95663-ho.karami]' is allowed to perform API calls: 0.0.0.0/0,::/0 2023-06-19 10:34:50,320 DEBUG [c.c.a.ApiServlet] (qtp1709366259-11:ctx-601ff485 ctx-d9ca0fd4) (logid:30a99e62) ===END=== 192.168.123.19 -- GET command=queryAsyncJobResult&jobId=c02aa83d-dd78-4922-9a0e-8915c4a73d3b&response=json&=1687158100241 2023-06-19 10:34:50,467 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-beb3c5d9) (logid:72557208) Execute sync-queue item: SyncQueueItemVO {id:382, queueId: 7894, contentType: AsyncJob, contentId: 510661, lastProcessMsid: 345051619551, lastprocessNumber: 20, lastProcessTime: Mon Jun 19 10:34:50 IRST 2023, created: Mon Jun 19 10:34:50 IRST 2023} 2023-06-19 10:34:50,471 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-beb3c5d9) (logid:72557208) Schedule queued job-510661 2023-06-19 10:34:50,486 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661) (logid:062ad570) Add job-510661 into job monitoring 2023-06-19 10:34:50,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661) (logid:c02aa83d) Executing AsyncJobVO {id:510661, userId: 39427, accountId: 19870, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAE2eAAAAAAAAmgMAAAAAAAABz3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345051619551, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Jun 19 10:34:50 IRST 2023, removed: null} 2023-06-19 10:34:50,501 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661) (logid:c02aa83d) Run VM work job: com.cloud.vm.VmWorkStop for VM 463, job origin: 510652 2023-06-19 10:34:50,510 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":39427,"accountId":19870,"vmId":463,"handlerName":"VirtualMachineManagerImpl"} 2023-06-19 10:34:50,525 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Stopped called on VM[User|i-943-463-VM] but the state is Expunging 2023-06-19 10:34:50,525 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":39427,"accountId":19870,"vmId":463,"handlerName":"VirtualMachineManagerImpl"} 2023-06-19 10:34:50,529 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Complete async job-510661, jobStatus: SUCCEEDED, resultCode: 0, result: null 2023-06-19 10:34:50,534 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Publish async job-510661 complete on message bus 2023-06-19 10:34:50,535 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Wake up jobs related to job-510661 2023-06-19 10:34:50,535 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Update db status for job-510661 2023-06-19 10:34:50,539 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661 ctx-c2a1bb00) (logid:c02aa83d) Wake up jobs joined with job-510661 and disjoin all subjobs created from job- 510661 2023-06-19 10:34:50,574 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661) (logid:c02aa83d) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 463, job origin: 510652 2023-06-19 10:34:50,574 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661) (logid:c02aa83d) Done executing com.cloud.vm.VmWorkStop for job-510661 2023-06-19 10:34:50,584 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-f056e924 job-510652/job-510661) (logid:c02aa83d) Remove job-510661 from job monitoring 2023-06-19 10:34:50,659 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652 ctx-546500c6) (logid:c02aa83d) VM state transitted from :Expunging to Expunging with event: ExpungeOperationvm's original host id: 193 new host id: null host id before state transition: null 2023-06-19 10:34:50,676 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652 ctx-546500c6) (logid:c02aa83d) Destroying vm VM[User|i-943-463-VM] 2023-06-19 10:34:50,676 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652 ctx-546500c6) (logid:c02aa83d) Cleaning up NICS 2023-06-19 10:34:50,716 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin java.lang.NullPointerException at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:528) at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:494) at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:2203) at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:2888) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy171.destroyVm(Unknown Source) at org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin.execute(DestroyVMCmdByAdmin.java:46) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-06-19 10:34:50,728 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Complete async job-510652, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530} 2023-06-19 10:34:50,732 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Publish async job-510652 complete on message bus 2023-06-19 10:34:50,732 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Wake up jobs related to job-510652 2023-06-19 10:34:50,732 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Update db status for job-510652 2023-06-19 10:34:50,736 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Wake up jobs joined with job-510652 and disjoin all subjobs created from job- 510652 2023-06-19 10:34:50,748 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Done executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin for job-510652 2023-06-19 10:34:50,748 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Remove job-510652 from job monitoring 2023-06-19 10:34:52,439 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cb3f5c89) (logid:2d3418e7) Begin cleanup expired async-jobs 2023-06-19 10:34:52,453 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cb3f5c89) (logid:2d3418e7) End cleanup expired async-jobs 2023-06-19 10:34:53,263 DEBUG [c.c.a.ApiServlet] (qtp1709366259-15:ctx-506fac14) (logid:2158719e) ===START=== 192.168.123.19 -- GET command=queryAsyncJobResult&jobId=c02aa83d-dd78-4922-9a0e-8915c4a73d3b&response=json&=1687158100242 2023-06-19 10:34:53,284 DEBUG [c.c.a.ApiServer] (qtp1709366259-15:ctx-506fac14 ctx-b71b52f6) (logid:2158719e) CIDRs from which account 'Acct[a27ca787-edb8-4361-b690-cb2f4cd95663-ho.karami]' is allowed to perform API calls: 0.0.0.0/0,::/0 2023-06-19 10:34:53,318 DEBUG [c.c.a.ApiServlet] (qtp1709366259-15:ctx-506fac14 ctx-b71b52f6) (logid:2158719e) ===END=== 192.168.123.19 -- GET command=queryAsyncJobResult&jobId=c02aa83d-dd78-4922-9a0e-8915c4a73d3b&response=json&=1687158100242

kiranchavala commented 1 year ago

@Hossein4185

Based on the exception 

2023-06-19 10:34:50,676 DEBUG [c.c.v.VirtualMachineManagerImpl] (API-Job-Executor-6:ctx-822bb171 job-510652 ctx-546500c6) (logid:c02aa83d) Cleaning up NICS

2023-06-19 10:34:50,716 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-6:ctx-822bb171 job-510652) (logid:c02aa83d) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin
java.lang.NullPointerException

It points to db discrepancy

Could you please check the status of the network and nic table

mysql> use cloud;

mysql> select * from nics where instance_id=463;

Identify the network the vm belongs to

mysql> select * from networks where name="<>"

Hossein4185 commented 1 year ago

@Hossein4185 Could you please send the complete management server for the date 2023-06-17.

You can upload it google drive and provide the link

in this day server log is very big file

kiranchavala commented 1 year ago

@Hossein4185 please provide the output of sql queries

Hossein4185 commented 1 year ago

mysql> select * from nics where instance_id=463\G 1. row id: 1825 uuid: 21b6149b-fc13-4935-80be-36e6612f1d67 instance_id: 463 mac_address: 06:0c:52:00:07:02 ip4_address: netmask: 255.255.255.0 gateway: 79.175.151.1 ip_type: Ip4 broadcast_uri: network_id: 205 mode: Dhcp state: Allocated strategy: Create reserver_name: DirectNetworkGuru reservation_id: NULL device_id: 0 update_time: 2023-06-19 11:33:41 isolation_uri: vlan://1222 ip6_address: NULL default_nic: 1 vm_type: User created: 2018-09-16 05:35:57 removed: NULL ip6_gateway: NULL ip6_cidr: NULL secondary_ip: 0 display_nic: 1 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ mysql> select * from networks where name = 'AF-CLD-INTR01'\G 1. row id: 204 name: AF-CLD-INTR01 uuid: 47608c16-ebb1-440e-9e9f-f5aee2b4887c display_text: Internet traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: gateway: cidr: mode: Dhcp network_offering_id: 22 physical_network_id: 201 data_center_id: 4 guru_name: DirectNetworkGuru state: Destroy related: 204 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: Domain network_domain: afra.cloud reservation_id: NULL guest_type: Shared restart_required: 0 created: 2017-08-29 11:40:39 removed: 2017-09-17 04:56:14 specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 2. row id: 205 name: AF-CLD-INTR01 uuid: 31746860-c146-494a-937b-717db41edc5e display_text: Internet traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: vlan://1222 gateway: cidr: mode: Dhcp network_offering_id: 22 physical_network_id: 201 data_center_id: 4 guru_name: DirectNetworkGuru state: Setup related: 205 domain_id: 1 account_id: 1 dns1: NULL dns2: NULL guru_data: NULL set_fields: 0 acl_type: Domain network_domain: afra.cloud reservation_id: NULL guest_type: Shared restart_required: 1 created: 2017-09-17 13:42:41 removed: NULL specify_ip_ranges: 1 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL 2 rows in set (0.00 sec)

Hossein4185 commented 1 year ago

Instance Expunge works correctly in another node, but it has an error in one node

kiranchavala commented 1 year ago

@Hossein4185

As a workaround, You can manually update the db records and then try to expunge the vm

mysql> use cloud mysql> update nics set state="Deallocating" , removed=now() where id=1825

DaanHoogland commented 1 year ago

@Hossein4185 you are using 4.13.1? this is quite an old version. If it is reproducible, please try to reproduce on latest (4.18 atm). A fix might already be made.

DaanHoogland commented 1 year ago

Sorry, what do you mean?

i-943-463-VM

Nothing happened

Hossein4185 commented 1 year ago

@Hossein4185 you are using 4.13.1? this is quite an old version. If it is reproducible, please try to reproduce on latest (4.18 atm). A fix might already be made.

I was using a one managament, but when I added a new management, there was a problem with the new management, but the previous management is working properly.

Hossein4185 commented 1 year ago

Sorry, what do you mean?

i-943-463-VM

Nothing happened

this is instance internal name

Hossein4185 commented 1 year ago

Whene I Destroyed instance on new management server, state filed on nics table has not changed from Allocated to Deallocating. I have to do this manually on the database

Hossein4185 commented 1 year ago

@Hossein4185

As a workaround, You can manually update the db records and then try to expunge the vm

mysql> use cloud mysql> update nics set state="Deallocating" , removed=now() where id=1825

I did it again and then restart the management solved the problem But this was done manually on the database. How can I solve the problem?

kiranchavala commented 1 year ago

@Hossein4185 are you able to reproduce this issue again

Can you try to launch a test vm and then expunge it ?

Hossein4185 commented 1 year ago

The VM launches correctly on New Management, but the expunge has a problem

DaanHoogland commented 1 year ago

@Hossein4185 this is quite an old version. Can you give the exact version of your installation (package version installed with yum or apt)? Maybe we can find out exactly what the NullPointerException means that way.

Hossein4185 commented 1 year ago

I use from this package:deb http://download.cloudstack.org/ubuntu bionic 4.13 and A few notes about the new Management: The Linux version that is currently working is Centos 6.10 final but I add Ubunu 18.04 as New Management and I didnt use from database ssl setting on new management node

DaanHoogland commented 1 year ago

yet another question: Can you update your linux systems to something newer? It all seems very old (linux and cloudstack)

from looking at the code for that version it seems the VM does not have a hypervisor type . Can you check that in your DB?

        final HypervisorGuru hvGuru = _hvGuruMgr.getGuru(vm.getHypervisorType()); // <== returns null

        s_logger.debug("Cleaning up NICS"); // <== seen in your logs
        final List<Command> nicExpungeCommands = hvGuru.finalizeExpungeNics(vm, profile.getNics()); // <== only serious NPE candidate is hvGuru
Hossein4185 commented 1 year ago

I added a management service with Centos7 and the problem was solved. It seems that if the database is configured with centos, it is not possible to add Ubuntu management to it.

DaanHoogland commented 1 year ago
  • Probably this is a bug on 4.13.1. Isn't it so?

might well be possible, but also it might be a limitation to use different MS OSses, though I thought that was possible. Glad it seems resolved.