search

ConPaaS-team / conpaas

ConPaaS: integrated runtime environment for elastic cloud applications
http://www.conpaas.eu
BSD 3-Clause "New" or "Revised" License
14 stars 3 forks source link

destroy_volume does too much Attempts #68

Closed FrancoCaffarraAndEsterDiBello closed 9 years ago

FrancoCaffarraAndEsterDiBello commented 9 years ago

Hello ConPaaS Team, we have found a strange slowness during the procedure to destroy a volume attached to a Virtual Machine. This makes very slow the stop procedure of services that use a lot of machines, each of them connected to an external volume. When we try to destroy a volume the function destroy_volume (in the core) does 11 iterations each of them requires 10sec sleep, even if the volume is correctly destroyed at first attempt ( we see that the image is deleted in ONE). This behaviour is the same in xtreemFs and Galera here the logs: xtreemfs:

2014-09-22 06:14:43,667 INFO conpaas.core.manager Detaching volume 80...
2014-09-22 06:14:43,667 DEBUG conpaas.core.controller detach_volume(volume=)
2014-09-22 06:14:43,837 INFO conpaas.core.manager Volume 80 detached
2014-09-22 06:14:43,838 INFO conpaas.core.manager Destroying volume with id 80
2014-09-22 06:14:43,838 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:14:43,848 INFO conpaas.core.manager Attempt 1: [ImageDelete] Cannot delete image. There are 1 VMs using it.
2014-09-22 06:14:53,858 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:14:53,905 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:14:53,930 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:14:53,952 INFO conpaas.core.manager Attempt 4: [ImageDelete] Error getting image [80].
2014-09-22 06:15:03,962 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:15:03,965 INFO conpaas.core.manager Attempt 5: [ImageDelete] Error getting image [80].
2014-09-22 06:15:13,966 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:15:13,969 INFO conpaas.core.manager Attempt 6: [ImageDelete] Error getting image [80].
2014-09-22 06:15:23,979 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:15:24,003 INFO conpaas.core.manager Attempt 7: [ImageDelete] Error getting image [80].
2014-09-22 06:15:34,008 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:15:34,011 INFO conpaas.core.manager Attempt 8: [ImageDelete] Error getting image [80].
2014-09-22 06:15:44,021 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:15:44,047 INFO conpaas.core.manager Attempt 9: [ImageDelete] Error getting image [80].
2014-09-22 06:15:54,057 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 06:15:54,084 INFO conpaas.core.manager Attempt 10: [ImageDelete] Error getting image [80].
2014-09-22 06:16:04,193 DEBUG conpaas.core.controller [delete_nodes]: killing iaas1044
2014-09-22 06:16:04,193 DEBUG ReservationTimer RTIMER removed node iaas1044, updated list []
2014-09-22 06:16:04,193 DEBUG ReservationTimer RTIMER Stopping timer for []
2014-09-22 06:16:04,193 DEBUG conpaas.core.clouds.base kill_instance(node=ServiceNode(id=iaas1044, ip=172.16.116.27))

Galera:

2014-09-22 12:56:29,668 INFO conpaas.core.manager Detaching volume 83...
2014-09-22 12:56:29,668 DEBUG conpaas.core.controller detach_volume(volume=)
2014-09-22 12:56:29,988 INFO conpaas.core.manager Volume 83 detached
2014-09-22 12:56:29,988 INFO conpaas.core.manager Destroying volume with id 83
2014-09-22 12:56:29,988 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:56:29,991 INFO conpaas.core.manager Attempt 1: [ImageDelete] Cannot delete image. There are 1 VMs using it.
2014-09-22 12:56:40,002 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:56:40,057 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:56:40,115 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:56:40,118 INFO conpaas.core.manager Attempt 4: [ImageDelete] Error getting image [83].
2014-09-22 12:56:50,126 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:56:50,129 INFO conpaas.core.manager Attempt 5: [ImageDelete] Error getting image [83].
2014-09-22 12:57:00,139 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:57:00,166 INFO conpaas.core.manager Attempt 6: [ImageDelete] Error getting image [83].
2014-09-22 12:57:10,177 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:57:10,180 INFO conpaas.core.manager Attempt 7: [ImageDelete] Error getting image [83].
2014-09-22 12:57:20,190 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:57:20,193 INFO conpaas.core.manager Attempt 8: [ImageDelete] Error getting image [83].
2014-09-22 12:57:30,203 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:57:30,227 INFO conpaas.core.manager Attempt 9: [ImageDelete] Error getting image [83].
2014-09-22 12:57:40,237 DEBUG conpaas.core.controller destroy_volume(volume=)
2014-09-22 12:57:40,263 INFO conpaas.core.manager Attempt 10: [ImageDelete] Error getting image [83].
2014-09-22 12:57:50,264 DEBUG conpaas.core.controller [delete_nodes]: killing iaas1049
2014-09-22 12:57:50,264 DEBUG ReservationTimer RTIMER removed node iaas1049, updated list ['iaas1051']
2014-09-22 12:57:50,264 DEBUG conpaas.core.clouds.base kill_instance(node=ServiceNode(ip=172.16.116.19, port=5555))
2014-09-22 12:57:50,267 DEBUG conpaas.services.galera.manager.config RemoveNodes node.id=iaas1049
tcrivat commented 9 years ago

Fixed in commit 7b44ac164844b71291dae2fe254da1ab4d47b7d8. This can be closed.