funccount, funclatency, stackcount user symbols don't work for containerized processes

[goldshtn]

All of these tools accept a symbol regex, which means they try to resolve the binary in which the symbol(s) may reside, and then try find all matching symbols (this happens before calling the actual attach API). There are two problems:

• The BPF.find_library helper doesn't take the target PID into account, so it passes pid 0 to bcc_procutils_which_so, which means it doesn't use the logic for looking into /proc/PID/maps to find the library path. (Which would be in the container's mount namespace, but still.)

• The underlying APIs for enumerating symbols (bcc_foreach_function_symbol) doesn't enter the container's mount namespace, so it fails to enumerate anything.

Just to clarify with a couple of examples:

# funccount jvm:*AllocTracer* -p $(pidof java)     # java is running in a container
unable to find library jvm      # caused by not looking in /proc/PID/maps to find libjvm.so path

# funccount c:open -p $(pidof java)
could not determine address of symbol open

# funccount c:*open* -p $(pidof java)
could not determine address of symbol _IO_file_fopen 

[tekumara]

I'm also experiencing this issue

Example when pid 4477 is running in a container:

$ sudo /usr/share/bcc/tools/funclatency '/tmp/lib_lightgbm3488589739562122259so:LGBM_BoosterPredictForCSR' -p 4477 -m
/tmp/lib_lightgbm3488589739562122259so
Traceback (most recent call last):
  File "/usr/sudo docker cp b17faa6f07f1:/tmp/lib_lightgbm3488589739562122259so /tmp/lib_lightgbm3488589739562122259soshare/bcc/tools/funclatency", line 215, in <module>
    pid=args.pid or -1)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 948, in attach_uprobe
    addresses = BPF.get_user_addresses(name, sym_re)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 892, in get_user_addresses
    BPF.get_user_functions_and_addresses(name, sym_re)])
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 908, in get_user_functions_and_addresses
    raise Exception("Error %d enumerating symbols in %s" % (res, name))
Exception: Error -1 enumerating symbols in /tmp/lib_lightgbm3488589739562122259so

My workaround for now is to copy the library from the container to the host:

sudo docker cp b17faa6f07f1:/tmp/lib_lightgbm3488589739562122259so /tmp/lib_lightgbm3488589739562122259so

[drandynisbet]

Hi, I'm having similar problems whilst trying to do some funccounts on USDTs in openjdk - is there any workaround for this container based use-case?

tplist -p $(pidof java) | grep gc /proc/6404/root/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so hotspot:mempoolgcbegin /proc/6404/root/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so hotspot:mempoolgcend /proc/6404/root/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so hotspot:gcbegin /proc/6404/root/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so hotspot:gcend

and funccount -p $(pidof java) u:/proc/6404/root/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so:gc__begin cannot attach uprobe, probe entry may not exist Failed to attach BPF to uprobe and

Trying with the mounted filesystem (presumably it is readonly) funccount -p $(pidof java) u:/var/lib/docker/aufs/mnt/7a9b8a8dce5f0e473b856984557903fe41084bccc1e32d48f63b8dbab0718598/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so:gc__begin USDT failed to instrument PID 6404

Thanks, Andy