gjcolombo
commented
1 year ago The instance state transitions above and in the logs look more or less correct to me, although I'm similarly at a loss as to why the saga table query didn't return the deletion saga.
It's currently expected that an Instance record will continue to have an "active" sled ID and Propolis ID even when the Propolis zone has been torn down; #2315 proposes to change that behavior. It otherwise looks like the system behaved as intended--Nexus eventually moved the instance to Stopped, and the deletion saga successfully ran to completion; after that point the instance's time_deleted should have been set and the instance should have appeared to any attempt to query it to be deleted.
The saga execution log for the saga that tried to re-create the instance, on the other hand, has this suspicious-looking entry:
{"msg":"recording saga event","v":0,"name":"nexus","level":20,"time":"2023-05-23T15:42:40.371974129-07:00","hostname":"oxz_nexus","pid":4365,"component":"SecStore","component":"nexus","component":"ServerContext","name":"496eabae-738d-4497-a183-1b75bf912c7c","event_type":"Failed(ActionFailed { source_error: Object {\"ObjectAlreadyExists\": Object {\"object_name\": String(\"sysbench-mysql-7\"), \"type_name\": String(\"disk\")}} })","node_id":"2","saga_id":"dec6089b-5291-42ca-ba84-6874c2dd44a0"}This suggests the already-extant object is a disk named sysbench-mysql-7 and not an instance. I don't know of a mechanism to ask Nexus to delete an instance's attached disks on instance deletion, so I think this is expected if the two instance create requests both asked to create a disk with this name.
@askfongjojo What mechanism did you use to try to re-create the instance? (Nexus doesn't appear to log the bodies of incoming HTTP requests, and I can't quite work out what value the "disks" parameter to the create request had in this case.) The instance creation API allows callers to specify whether they want to create a new disk or merely attach an existing one (in the InstanceDiskAttachment component); if you asked to "create" the disk both times, switching the re-creation attempt's disk request type to "attach" might resolve the issue.
askfongjojo
davepacheco
This happened when I used a script to invoke
oxide instance stopfor 10 instances successively in a loop, and then delete them with cli in another loop. The requests came back with success response (ignore the two destroyed instances - they will be covered in another ticket):However, these instances are now "shell" records in the instance table - the propolis zones are already gone from the sled:
I looked into one of the instances, sysbench-mysql-7 which is located on sled gc11. It was set to
stoppedat 22:40:03The zone was removed at 22:40:08 according to the sled-agent log for gc11:
but the delete saga was created 44 seconds later, at 22:40:52, presumably because nexus was busy taking deletion requests of other instances:
I didn't find any saga error in the log but the saga also doesn't exist in the database:
At 22:42:40, I tried to recreate a VM with the same name but that failed with an
object already existserror. These VMs are now ghost records and cannot be removed because the VMs don't have a backend and can't be stopped/deleted.The complete nexus log is located in catacomb.eng.oxide.computer:/data/staff/dogfood/may-23/.