input-output-hk / nixops-packet

NixOps Packet.net Plugin
GNU Lesser General Public License v3.0
5 stars 5 forks source link

"waiting for SSH...." forever, even if provisionin fails #16

Closed grahamc closed 3 years ago

grahamc commented 4 years ago
packet-spot-eval-3...........> waiting for SSH.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................1.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

but about half way through provisioning failed and the server was reaped.

johnalotoski commented 3 years ago

The "waiting for SSH" output during a new provision should only be seen after the provision has succeeded as reported by the wait_for_state(target_state) method. So presumably something has gone wrong in the provisioning after the on-instance provisioner and packet.net have already reported provisioning success.

I think, like other plugins in this case, the remedy would be the typical --check option to make sure the instance is still really there, followed by deploy with --allow-recreate if not.

I'm also considering adding a packet specific wait_for_ssh method with a modified callback that checks packet instance health every X seconds (ex: 10) while waiting for ssh so that if a reaping occurs, the wait_for_ssh function won't continue indefinitely.

johnalotoski commented 3 years ago

Fixed in https://github.com/input-output-hk/nixops-packet/commit/e9e37e3d71aed584f46fecb2b631173a567e8249.

There is now an instance API health check at the start of a wait_for_ssh_nixops_packet function call and every 60 seconds after.

johnalotoski commented 3 years ago

Closing. Re-open this ticket or create a new ticket if the issue persists.