search

spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
491 stars 43 forks source link

--instance-initiated-shutdown-behavior terminate #122

Open turian opened 1 year ago

turian commented 1 year ago

I want to 'spotty stop' my instance after 'spotty run' is down. This saves money of course. But I don't know how.

When run panes are over, you are stuck in a tmux screen with dead pane. I hoped I would be dropped back into my local shell and the next step of my script after 'spotty run' would be 'spotty stop'. However, this doesn't work.

How can I spotty start the instance with --instance-initiated-shutdown-behavior terminate? Then if 'shutdown' is the last command of run, I can achieve the desired bevahior.

Can you propose a simple effective workaround?

turian commented 1 year ago

I tried running /sbin/shutdown -h now as the last step of my spotty scripts that I spotty run, but I get the following:

/sbin/shutdown -h now
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
Failed to talk to init daemon.

Pane is dead (status 0, Mon Dec 19 17:51:00 2022)
[spotty-sc0:train_name*

How can I get the pane actually to terminate, so my outer script (which does spotty run then spotty stop) can continue?