neoave / mrack

Multicloud use-case based multihost async provisioner for CIs and testing during development
Apache License 2.0
11 stars 14 forks source link

fix: do not end provisioning if there are no resources and handle premature deletion #137

Closed Tiboris closed 3 years ago

Tiboris commented 3 years ago
fix: do not end provisioning if there are no resources

Providers ended provisioning early by throwing an exception
ValidationError which caused mrack to end provisioning
even when retry strategy has been enabled.
Now we create error Host object to continue provisioning with.
Also fixed condition when we need to cleanup the
error hosts for the next retry of the provisioning.

fix(Virt): Create more readable output and move to debug

fix(Beaker): delete_host handle premature deletion

fix(Podman): handle premature deletion

Podman now handles premature deletion and
we fixed missing max_retry arg in init()
Tiboris commented 3 years ago

this might help to review:

diff --git a/src/mrack/providers/provider.py b/src/mrack/providers/provider.py
index b8a827a..63bcd12 100644
--- a/src/mrack/providers/provider.py
+++ b/src/mrack/providers/provider.py
@@ -131,7 +131,7 @@ class Provider:
         return res, host

     async def _provision_base(
-        self, reqs, res_check_timeout=60, res_busy_sleep=10
+        self, reqs, res_check_timeout=0.60, res_busy_sleep=0.10
     ):  # pylint: disable=too-many-locals, too-many-branches, too-many-statements
         """Provision hosts based on list of host requirements.

@@ -154,7 +154,7 @@ class Provider:

         error_hosts = []
         res_check_start = datetime.now()
-        while not await self.can_provision(reqs):
+        while True or not await self.can_provision(reqs):
             if datetime.now() - res_check_start >= timedelta(minutes=res_check_timeout):
                 # create error host object so retry strategy can continue
                 # instead of throwing exception to fail at once without retry