systrace and syscount

[goldshtn]

systrace would be an strace-like tool (with limited parsing of arguments). It would provide a general overview of process syscall activity, which can then be examined in more detail with argdist/trace or dedicated tools.

syscount would provide syscall statistics on the whole system or a specific process. Should probably mimic syscount from Brendan Gregg's perf-tools.

[goldshtn]

In terms of implementation, both of these tools would have to use ^[Ss]y[Ss]_* kprobes because the syscall tracepoints don't work at this time. (See #748 for discussion.)

[vitalyvch]

Just for note: I'm going to release something similar but with accent on performance. It is written in pure C.

[vitalyvch]

Guys, what do you think about idea of samples/bpf/tracex5_*.c in linux kernel sources? They did something similar, but by kprobing seccomp_phase1(). It allow them to load much much faster @4ast, @brendangregg

[goldshtn]

Can't say I love the idea of installing a dummy seccomp program and then probing an internal seccomp function. This may be quite brittle. What I'd really want is syscall tracepoints. But if this approach works more or less reliably on all kernels from, say, 4.1 -- then maybe ...

[brendangregg]

@vitalyvch I haven't hacked on this much, but how do we even enable seccomp for the entire system? The tracex5 example seems to only enable seccomp for child processes.

Enabling seccomp just to trace syscalls does seem an odd way to do this.

[vitalyvch]

Thanks. I'm not very familiar with seccomp so your answers are really useful for me. I already have added preliminary tracepoint support to my tool, and think about adding that seccomp trick.

[goldshtn]

@vitalyvch Is your tool something that you plan to release publicly? Is there a likely timeframe? I ask because systrace/syscount can already be immensely useful, and if you think there's a long way to go, we can implement something temporary in the interim.

[goldshtn]

@vitalyvch Have you been able to make progress on releasing your tools? If not, I think we should consider implementing simple versions of those in Python. @brendangregg

[vitalyvch]

@goldshtn, I'm sorry for delayed reply. So, I do not stop developing. Yes, there is a plan to release, but I should discuss details with team. Currently I'm working on imitating follow-fork option.

[marcinslusarz]

I'm working with Vitalii on this project. It's basically a BSD-licensed strace replacement written in C using bcc/ebpf with both binary and text output. I can't give you the exact timeline, but the plan is to release it in a few months. Initially it will be a part of pmem/nvml, but since it will not depend on anything in nvml (except maybe on testing framework), we might push it to separate repository. Personally I'd be happy to include it in bcc tree...

We'll let you know here when the tool will be ready in at least experimental state.

[vitalyvch]

So, since last commits, tracing sys* and SyS* does not produce the desired result: #940

@goldshtn

[goldshtn]

@vitalyvch What do you mean by "since last commits"? Did you bisect this to find the latest commit in which it worked?

[vitalyvch]

I didn't bisect. I just install night builds every morning, and do manual testing every few days.

On Friday tests failed. Most likely something unexpected was introduced on Wednesday or Thursday.

@goldshtn

[goldshtn]

Confirmed, I see this on the latest master. Trying to bisect.

[goldshtn]

All right, so it worked here: 199b341ac71d9bee9ebbcaa2908647ff45c5c394 Doesn't work here: 4a57f4dd7696ec226592ad72c1398bf3cd770383 Only a couple of more to go.

[goldshtn]

OK, so looks like it was broken by this merge: b77915df95b971557dcb42606f493ffb5175a8ec

[goldshtn]

Going to file this on #940.

[vitalyvch]

Some update:

Our current code is available here: https://github.com/vitalyvch/strace.ebpf

Tools is not released, but could be interesting for you.

@goldshtn , @brendangregg