Correlating sessions between auditd and SSM

[patdowney]

Hi folks, I've got a query about trying to tie together the SSM session logs and linux auditd sessions/events.

I had hoped that, from taking a look at the process tree, if I am able to record invocations of /usr/bin/ssm-session-worker then I would be able to get a correlation between the SSM session identifier and an auditd session, and use that to be able to match auditd session to a particular ssm session, even if multiple AWS principals are logged in at the same time. Unfortunately it doesn't work that way, and to my knowledge I can't tell auditd to monitor a process and all it's children.

It looks as though the session identifier that is logged against all these commands (both ssm-session-worker and all it's children) is ses=4294967295, which along with auid=4294967295 is not much good to me. From what I can see is 4294967295 is effectively -1 or unset, and matches many system processes. And because the auid and ses are useless, tracking sessions across sudo invocations is almost impossible. I've shown an example below:

For the following I used these auditd rules:

-w /home/ssm-user/bin/session-one -p x -k github-issue
-w /home/ssm-user/bin/session-two -p x -k github-issue
-w /home/ssm-user/bin/session-one-sudo -p x -k github-issue
-w /home/ssm-user/bin/session-two-sudo -p x -k github-issue

This is the process tree showing two distinct ssm sessions

root      1078  *snip*   0:00  \_ /usr/bin/ssm-agent-worker
root      1334  *snip*    0:00      \_ /usr/bin/ssm-session-worker botocore-session-1680215784-07473558e6a50c0ce
ssm-user  1346  *snip*   0:00      |   \_ sh
root      1347  *snip*    0:00      \_ /usr/bin/ssm-session-worker botocore-session-1680215784-0d76699d8d14b4279
ssm-user  1362  *snip*    0:00          \_ sh

These are the sys calls logged

type=SYSCALL msg=audit(1680216640.982:110): arch=c00000b7 syscall=221 success=yes exit=0 a0=34c4e870 a1=34c4e750 a2=34c2abe0 a3=4ffdd0 items=3 ppid=1346 pid=1365 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=pts1 ses=4294967295 comm="session-one" exe="/usr/bin/env" key="github-issue"
type=SYSCALL msg=audit(1680216647.292:111): arch=c00000b7 syscall=221 success=yes exit=0 a0=36c81870 a1=36c81750 a2=36c5dbe0 a3=4ffdd0 items=3 ppid=1362 pid=1367 auid=4294967295 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=pts2 ses=4294967295 comm="session-two" exe="/usr/bin/env" key="github-issue"
type=SYSCALL msg=audit(1680216767.731:122): arch=c00000b7 syscall=221 success=yes exit=0 a0=cbbffd0 a1=cbc67c0 a2=cbc16b0 a3=4ffdd0 items=3 ppid=1394 pid=1413 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2 ses=4 comm="session-two-sud" exe="/usr/bin/env" key="github-issue"
type=SYSCALL msg=audit(1680216779.961:123): arch=c00000b7 syscall=221 success=yes exit=0 a0=f93320 a1=f1fc30 a2=f236b0 a3=4ffdd0 items=3 ppid=1376 pid=1416 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=3 comm="session-one-sud" exe="/usr/bin/env" key="github-issue"

You'll note that in all of them, the non-sudo calls share the same auid and ses even though they're in different SSM sessions (the calls under sudo get issues 'proper' auid and ses values, but no way to map them to SSM). And because I'm not logging all execve sys calls, I can't even use ppid to pid mapping to associate the commands run under sudo to their parent processes (which would also be a pain to analyse).

What I'd love would be better integration between the SSM agent and the linux auditing subsystem. But I'd also love to understand if I'm doing it wrong (e.g. are there some PAM settings I should be using) and how other people are correlating sessions, or is everyone satisfied with the current SSM session manager logging even though it's nowhere near as rich?

[patdowney]

For ref - I think gravitational/teleport implements a similar capability - https://github.com/gravitational/teleport/issues/13785

[yuting-fan]

@patdowney - Session Manager worker logs are intended for troubleshooting purpose, not for auditing purpose. Please see the documentation page.

You may use CloudTrail to identify which IAM identity started which session, and the session ids in CloudTrail events match the session worker ids when the sessions are still connected. The session worker threads are terminated when the session is terminated. However, please bear in mind that CloudTrail gives you the auditability of the caller of StartSession, instead of what OS operations are performed on the host.

Hope this clarifies!