Ec2 has not been terminated

[pharindoko]

Hey,

I have this rare case that an ec2 is still available for days.

• Systemcheck shows that the instance is not reachable anymore.
• Github workflow succeeded
• Stepfunction workflow succeeded

aws console:

system-log

Anyone had the same issue ?

[kichik]

Anything of interest in /var/log/cfn-cmd.log, /var/log/cfn-cmd-init.log and the output of dmesg?

[pharindoko]

Anything of interest in /var/log/cfn-cmd.log, /var/log/cfn-cmd-init.log and the output of dmesg?

Not really - could I see some of that stuff in the cloudwatch logs. I'm unable to connect to the instance via ssm.

[kichik]

Those logs do not go in CloudWatch.

It's strange that the step function completed but the system log doesn't show the aws stepfunctions send-task-success call. Whenever I see something like this, I immediately assume OOM. That's why I asked for dmesg. But it's very strange for that to happen so fast that the log would only show one heartbeat (should be one every minute) and nothing else. Even the runner log itself seems truncate. Is it just as terse in CloudWatch?

[pharindoko]

I'll check what I can find in cloudwatch. It's a super rare case but appeared now twice in one week. Maybe I need an additional watchdog. This is why I'm asking if anyone else had that problem.

[pharindoko]

Related to #537