iterative / terraform-provider-iterative

☁️ Terraform plugin for machine learning workloads: spot instance recovery & auto-termination | AWS, GCP, Azure, Kubernetes
https://registry.terraform.io/providers/iterative/iterative/latest/docs
Apache License 2.0
288 stars 27 forks source link

`cml-runner` terraform not available in host machine #603

Closed DavidGOrtega closed 2 years ago

DavidGOrtega commented 2 years ago
Error: Error checking the runner status

  on main.tf line 49, in resource "iterative_cml_runner" "runner2":
  49: resource "iterative_cml_runner" "runner2" {

-- Logs begin at Sun 2022-06-05 18:51:13 UTC, end at Sun 2022-06-05 18:53:59
UTC. --
Jun 05 18:53:54 iterative systemd[1]: Started cml.service.
Jun 05 18:53:56 iterative cml.sh[15532]: {"level":"error","message":"terraform
version\n\t\n\t/bin/sh: 1: terraform: not found\n","stack":"Error: terraform
version\n\t\n\t/bin/sh: 1: terraform: not found\n\n    at
/snapshot/cml/src/utils.js:20:27\n    at ChildProcess.exithandler
(node:child_process:406:5)\n    at ChildProcess.emit (node:events:527:28)\n
at maybeClose (node:internal/child_process:1092:16)\n    at Socket.<anonymous>
(node:internal/child_process:451:11)\n    at Socket.emit
(node:events:527:28)\n    at Pipe.<anonymous>
(node:net:709:12)","status":"terminated"}
Jun 05 18:53:56 iterative cml.sh[15532]: {"level":"info","message":"waiting 10
seconds before exiting..."}
dacbd commented 2 years ago

I feel like there was a one-off setup script issue? are you able to reproduce, and get the cloud-init/setup type logs?

DavidGOrtega commented 2 years ago

I feel like there was a one-off setup script issue? are you able to reproduce, and get the cloud-init/setup type logs?

Yes I was able to reproduce it that day constantly, but not now... It must be the ubuntu distro that is loaded... I can confirm that was not a terraform package issue because AWS and GCP worked fine

DavidGOrtega commented 2 years ago

Unable to reproduce. Have tried many times.