garloff
commented
7 months ago This all is only relevant if we use diskless flavors, of course.
One idea would be store a list of volumes prior to the VM creation call. Then, IF N>0 VMs failed to come up (and maybe after we have done the name assignment), check what volumes exist and if there are extra volumes (not belonging to any of the created VMs and not preexisting) and has the expected image metadata, it's very likely a volume that should be cleaned up.
After addressing #146 with #147, we still see leaked volumes. Here's what I observe: (1) VMs fail to be created with nova noting
| fault | {'code': 500, 'created': '2024-02-06T18:58:12Z', 'message': "Build of instance 52fde924-35b1-4d8d-ac1c-2acfb262bdfa aborted: Connection to cinder host failed: Unable to establish connection to https://prod3.api.pco.ps-intern.de:8776/: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without "} |which means that cinder did not report a successful created volume back to nova. (2) Occasionally volume name assignments fail, as does volume deletion (JumpHost). And I have even seen a failed volume list.Interpretation: (2) Shows me that talking to the cinder API from the VM does also fail occasionally, just like the communication when approached from nova. The errors here do not directly lead to volume leaks. The failed deletions are retried. The failed renames do not cause leakage either, as we still try to delete by ID.
(1) is what causes the leaks. Maybe more often than failed cleanups from nova. The fact that cinder does not report success back to nova does not mean that the volume is not created. If it is, we have nothing that cleans it up. nova does not track it nor does
api_mon.sh. This is not easy to address ...