search

GaloisInc / BESSPIN-FETT-Portal

The web-based portal used by FETT Researchers to manage Target instances.
Apache License 2.0
0 stars 0 forks source link

Investigate why ThunderingTurn instance was left in terminating status #446

Open kurthopfer opened 4 years ago

kurthopfer commented 4 years ago

https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/fett.log/log-events/i-03aca754146af54ec

It appears that FETT sent the success message to the status queue and it was triggered to update, investigate the lambda and validate why the db wasnt updated.

mattlebeau-galois commented 4 years ago

~2 hours of investigation ~2 hours implementation of fix

alex-fivetalent commented 4 years ago

@kurthopfer It seems the messages were received from the fett target by the termination lambda and there is a log that noted the db was being updated to terminated. The instance was shutdown internally. I think it somehow got set back to terminating from terminated. There is no log of it getting simultaneously shutdown by the auto cleanup lambda or the force termination lambda. The only odd thing I see is that the termination lambda received two almost simultaneous requests. The second one erred out. I have tried some fancy clicking on the portal but can't seem to replicate a double terminate signal. Could be an edge case I can't place. For now we can just check in and make sure it doesn't happen again. Its been almost two weeks with no repeat.