Closed gpierre42 closed 9 years ago
This is the same as issue #68. The service will stop, but only after a long time.
Really? I had time to go and have lunch, and when I came back the service still hadn't stopped. In case you wonder why the instances are now gone, I killed them by hand in the EC2 dashboard.
That's what I remember was happening. I will check again myself.
Indeed, this is a different issue than #68. The service doesn't stop at all. But it happens only on Amazon EC2. I tested on OpenNebula, and the service stops fine there (after a delay of ~2 min because of #68). I will update the title to reflect that this is an Amazon EC2-specific issue.
Is there AMI image we can use?
The current ConPaaS deployment on Amazon EC2 is accessible here ( http://conpaas-online.ddns.net/ ) and it uses ami-5594d765. You can use directly the installation located at that link.
I remembered it was posted somewhere before :) However, I can not find the AMI under community AMIs on EC2, I will try will try the direct link, thanks.
This happens because the scratch volume is not detached, although the detach_volume function from the EC2 driver library returns True. Because the volume is not detached, deleting the volume fails and an exception is thrown in this line:
https://github.com/ConPaaS-team/conpaas/blob/dev/conpaas-services/src/conpaas/core/manager.py#L254
The exception is not caught anywhere in the Galera manager code, so the whole service stopping procedure halts.
This exception is not caught in XtreemFS either, but in the XtreemFS case, the volume is detached successfully, so this bug does not happen.
A quick (and dirty) fix would be to suppress that exception. The service will shut down promptly (including the agent VM), but the volume will not be deleted.
According to the Amazon EC2 documentation, a volume should be unmounted from the VM before detaching:
http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DetachVolume.html
As the documentation says: "Failure to do so will result in the volume being stuck in "busy" state while detaching.". This seems to be exactly what is happening in our case.
The commit that added support for volumes in the Galera service (db53252c5472f0d905f12122e83cb31f08c86642) does not seem to contain any code that does the unmount, so I presume that Galera does not unmount the volume at all.
There is also another possibility, to forcefully detach the volume even if it is not unmounted first. This is allowed in the Amazon EC2 CLI, however the libcloud library that we use does not seem to support this option:
So, as a conclusion, this issue can be solved only by modifying the Galera service to unmount the volume before detaching.
How we can delete the instances of a frozen service? Someone can do this for us?
Sure, I can do that. Any time you need me to do this, please send me an e-mail on my mail address ( teodor.crivat@gmail.com ). Thanks.
We have upgrade the code for the unmount in a7579101d693ad9cf56551b42ab87a38f8b508c8. Please can you upgrade the installation with the current code to test it?
It fails with "Sorry: IndentationError: unexpected indent (role.py, line 391)". I fixed the indentation and will try again.
ok thanks :)
Il giorno 11/ott/2014, alle ore 19:37, Teodor Crivat notifications@github.com ha scritto:
It fails with "Sorry: IndentationError: unexpected indent (role.py, line 391)". I fixed the indentation and will try again.
— Reply to this email directly or view it on GitHub.
It works! The Galera service now stops cleanly and promptly on both Amazon EC2 and OpenNebula.
Thanks @FrancoCaffarraAndEsterDiBello for your prompt action.
This issue can be closed now.
I tried using the Galera service on the EC2-based ConPaaS deployment provided by @tcrivat. I created the service, then started it, then stopped it. For some reason the service didn't stop, and remained in "adapting" state forever.