iterative / terraform-provider-iterative

☁️ Terraform plugin for machine learning workloads: spot instance recovery & auto-termination | AWS, GCP, Azure, Kubernetes
https://registry.terraform.io/providers/iterative/iterative/latest/docs
Apache License 2.0
290 stars 27 forks source link

`runner` does not seem to destroy the resource if fails #348

Closed DavidGOrtega closed 2 years ago

DavidGOrtega commented 2 years ago

It clears up the resource if failing creating the machine, however, if registering the runner fails it might not be destroying the resource!?

dacbd commented 2 years ago

I feel adding something to the effect of systemctl is-active cml.service (returning "failed") in here: https://github.com/iterative/terraform-provider-iterative/blob/b47f8014c1a222362b43ac9c5afc8a9522fa7ea3/iterative/resource_runner.go#L253-L271

when it fails, triggers a cleanup, capturing https://github.com/iterative/cml/issues/906 as well. In the latter's error printing coming from this condition: https://github.com/iterative/terraform-provider-iterative/blob/b47f8014c1a222362b43ac9c5afc8a9522fa7ea3/iterative/resource_runner.go#L278-L284

Thoughts?

dacbd commented 2 years ago

I believe this was resolved with: https://github.com/iterative/terraform-provider-iterative/pull/460 I tried to reproduce this with: https://github.com/iterative/cml/compare/trigger-tpi-348

DavidGOrtega commented 2 years ago

closed via #460