search

ix-project / ix

IX a protected dataplane operating system project
http://ix-project.github.io/ix/
Other
127 stars 39 forks source link

Profiling in IX #35

Open ash995 opened 3 years ago

ash995 commented 3 years ago

How would we profile applications running on top of IX and the IX code itself using profilers like oprofile and perf? We did try using perf to record events against the IX pid but the recorded samples show no symbols related to our application code or IX code.

prekageo commented 3 years ago

What symbols do you see? What do you see if you run perf script? IX runs in non-root mode. So, you have to profile the guest and pass to perf the ix binary as the guest kernel.

ash995 commented 3 years ago

We see find_numasocket.isra.2 at the top of the profile consuming some 30 to 40 percent of the cycles. All the other symbols we see are kernel symbols.

So, are you suggesting we use perf kvm to profile IX and and our application code? If not so, it would be helpful if you could point us to the specific command and also how would we pass the IX binary to the perf profiler in a way such that the IX symbols are exposed to the perf profiler.

prekageo commented 3 years ago

Did you check the output of perf script? Have you read https://man7.org/linux/man-pages/man1/perf-record.1.html and https://man7.org/linux/man-pages/man1/perf-kvm.1.html? Specifically, take a look at the --guestkallsyms option.

ash995 commented 3 years ago

We ran the following command

sudo perf kvm --guest --guestkallsyms=/home/fred/ix/dp/ix record -e cycles --pid 169253 sleep 15

Here pid is the IX process id as shown in top.

The output did change but now we see just addresses instead of actual symbols.

Screenshot from 2021-03-10 21-43-28

Also, we get the following warning when running the profiler.

Screenshot from 2021-03-10 21-48-42

prekageo commented 3 years ago

You are passing the wrong parameter in guestkallsyms. You should not pass the binary but its symbols in a format compatible to /proc/kallsyms.

prantosky commented 3 years ago

We have generated the symbols for IX, using the following command: nm -anC ~/ix/dp/ix | c++filt > ix_syms

We think we get the same output as the format of /proc/kallsyms, the symbol file is attached.

We then start the recording using the following command, sudo perf kvm --guest --guestkallsyms=~/ix_syms record -e cycles --pid <pid>

Later, after sudo perf kvm report, we get the following output,

this output

Is there anything wrong with this procedure?

prekageo commented 3 years ago

It's been many years since I dealt with it but... why do you use --guestkallsyms on record? It makes sense to use it with report.

prantosky commented 3 years ago

We did try --guestkallsyms on record as well as report, still got the same output.