Feature: post-mortem analysis

[topimiettinen]

Now, after the firejailed application has exited, Firejail just performs necessary cleanup of the file system and exits. But before doing that, it could be interesting (optionally) to see various statistics and analyze the changes which the application made on the file system:

• display IP packet counters (reuse IP firewall packet counter BPF code from systemd)
• list files in temporary or overlay file systems (run /bin/ls there), show changes for copied files (diff), or maybe just make the file system available for use of other tools (mount and launch a shell in the file system)
• show summary of use of system calls (ptrace) for seccomp fine tuning, seccomp hits/misses (with Linux 5.0 SECCOMP_RET_USER_NOTIF?)
• report which Firejail rules were used and which were not (could need FUSE, ptrace etc + considerable amount of code)

Also it could be useful to be able to stop the application, do some analysis and then continue running.

[netblue30]

Let's do it! I moved upcoming 0.9.62 release on a separate branch for fixes only (release-0.9.62). A release is pending, probably next week. Use master branch for the development.

[topimiettinen]

Should be actually almost doable with existing sandbox joining features. The workflow would be something like this:

1. Launch a shell with the same profile

   firejail --noprofile --private-home=.lesshst --name=less  --net=none bash

2. Launch the app, joining the shell sandbox

   firejail --join=less less /etc/passwd

   (searching adds entries to .lesshst, for example "root")

3. Post-mortem with the shell launched in 1., some help needed from firejail

   $ cat .lesshst
   .less-history-file:
   .search
   "join
   .search
   "root
   $ sudo diff .lesshst /proc/2650/root/home/topi/.lesshst 
   3a4,5
   > .search
   > "root
   $ ip -s link
   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   RX: bytes  packets  errors  dropped overrun mcast   
   0          0        0       0       0       0       
   TX: bytes  packets  errors  dropped carrier collsns 
   0          0        0       0       0       0       

   Here sudo diff and the path should be replaced with something nicer like firejail --join=less --diff, or making also original filesystem for /home available somewhere for the shell in 1. for manual diffing.

But I have some problem here (probably my configuration) that joining does not work (so 2. above is a bit fake), instead I get this error:

Switching to pid 2650, the first child process inside the sandbox
Error: no valid sandbox

Strace shows this:

 2749  openat(AT_FDCWD, "/proc/2650/root/run/firejail/mnt/ready-for-join", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)

I've disabled kernel.yama.ptrace_scope.

Of course, the workflow in 2. could be simply to reuse the shell launched in 1. to start the app like I did. But then the sandbox contains a shell, which is not always wanted.

[netblue30]

Switching to pid 2650, the first child process inside the sandbox Error: no valid sandbox

I think It needs a delay, maybe the kernel didn't set up all the namespaces yet, or it didn't get a chance to update /proc filesystem?

I played a little bit with your idea of --join and put some code in main.c (look at line 151 in the file). Uncomment the line and run from a terminal:

$ firejail --net=eth0 firefox --no-remote

It will start a second sandbox with a tcpdump (the code is around line 2754), it will join the network namespace of the first sandbox, and it will print the namespace traffic in the terminal. Quite funny, when I close firefox the two sandboxes are closed nicely, but tcpdump survives and it moves to systemd for a parent. Assuming we replace tcpdump with one of our programs, we can close this one also after printing whatever data we need to print on the terminal.

[smitsohu]

I think It needs a delay, maybe the kernel didn't set up all the namespaces yet, or it didn't get a chance to update /proc filesystem?

Maybe we could finally add a delay to the join option, as has been requested already in #2139. If it is helpful I can send a fix.

tcpdump survives and it moves to systemd for a parent

For external tools one could also consider PR_SET_PDEATHSIG, which is preserved across execve.

[netblue30]

Maybe we could finally add a delay to the join option, as has been requested already in #2139. If it is helpful I can send a fix.

Yes, let's do it. I forgot about #2139!

Thanks for PR_SET_PDEATHSIG, I'll try it.

[mYnDstrEAm]

Would be amazing if things like this could be logged.

Such logs could later and at real-time even be analyzed by intrusion detection systems. Would be especially useful if combined with other info such as of ps and syslog entries related to the running process. This could advance IDSs by associating made changes with processes and also tracking other stats (in a way that's useful).

Of course one would need to make sure that the logs are short enough summaries, can't grow too large (e.g. configurable automatic moving of old logs to external media), don't take too long to create (i.e. don't prepare the log after the log after the app has exited but update the log every few minutes so even if the logging crashes there are still some logs and it'll be fast to create the complete log when it exits) and can be nicely read by other software.