BPFTrace Uprobe Support

[chengruizhe]

Is your feature request related to a problem? Please describe. The current bpftrace feature allows attaching kprobes to kernel functions, but not to any user space functions.

Describe the solution you'd like BPFTrace support for deploying custom uprobes to user space functions. This would enable dynamic tracing of any user space applications or libraries. It should be able to deploy to all the nodes with the target UPID and tracepoint.

[mayanksingh2298]

Hi.. this seems interesting. I myself have been trying to use uprobes with bpftrace but there isn't much documentation around using this with PXL scripts. What is the status on this?

[ddelnano]

Hi @mayanksingh2298, the current bpftrace support does not have support for uprobes today. The goal with the issue was to get feedback from the community and understand how much interest there is.

I would be very interested to hear more about your use case for this feature. Can you summarize what types of uprobes / problems you are trying to solve with this?

[mayanksingh2298]

Thanks for replying @ddelnano. I'm working on building a live debugger - starting with go lang. Something like delve with their ebpf backend.

[mayanksingh2298]

@JamesMBartlett mentioned in slack that there is some undocumented support for uprobes in PXL. So I was wondering if how much work has been done in pixie around this.

[mighty1231]

I don't know how the pixie integrates with bpftrace much. I may need some help for it, anyway let me share my opinion for uprobe. In host node, running bpftrace in specific namespace can be done with following script...

podman run --rm --pid=ns:/proc/TARGET_PID/ns/pid --privileged \
  -v /sys/kernel/debug:/sys/kernel/debug:ro \
  -v /usr/src:/usr/src:ro \
  quay.io/iovisor/bpftrace:v0.19.1 \
  bpftrace -e 'uprobe:/proc/1/root/some_path/for/binary_or_library_in_container:some_function { @[probe]++ }'

Explanation for /proc/1/root/

Here's a question. Isn't it possible to uprobe/uprobe like above script? Pixie may help to configure TARGET_PID from upid. However, commit 4ba8d065408d6d37d3ce491075730529a6fbbaac shows the bpftrace script without binary file path, it seems to prohibit to use uprobe on dynamic library instead of main program.

[josyulav]

/ import pxtrace import px

Use this scratch pad to write and run one-off scripts.

If you switch to another script, refresh, or close this browser tab, this script will disappear.

program =""" /*

• threadsnoop - List new thread creation.
• See BPF Performance Tools, Chapter 13, for an explanation of this tool.
• Copyright (c) 2019 Brendan Gregg.
• Licensed under the Apache License, Version 2.0 (the "License").
• This was originally created for the BPF Performance Tools book
• published by Addison Wesley. ISBN-13: 9780136554820
• When copying or porting, include this comment.
• 15-Feb-2019 Brendan Gregg Created this. */

include"/home/ubuntu/victim.h"

BEGIN { printf("%-15s %7s %-16s %s %11s\n", "TIME", "PID", "COMM", "FUNC","ARGS"); } / uprobe:/home/vjosyula/work/bpftime_enunomia/building_from_source/bpftime/example/minimal/victim:target_func1, { $comp = (struct composite *)arg0; printf("%15s %7d %-16s %s %d %d %d\n", strftime("%H:%M:%S.%f", nsecs), pid, comm, func,$comp->a,$comp->b,$comp->c); } """

def sample_uprobes_application_func(): table_name = 'sample_uprobes_application' pxtrace.UpsertTracepoint('sample_application_tracer', table_name, program, pxtrace.uprobe(), "10m")

df = px.DataFrame(table=table_name)

px.display(df)
return df

{
    "variables": [],
    "globalFuncs": [
        {
            "outputName": "results",
            "func": {
                "name": "sample_uprobes_application_func",
                "args": []
            }
        }
    ],
    "widgets": [
        {
            "name": "Results",
            "position": {
                "x": 0,
                "y": 0,
                "w": 12,
                "h": 5
            },
            "globalFuncOutputName": "results",
            "displaySpec": {
                "@type": "types.px.dev/px.vispb.Table"
            }
        }
    ]
}

Bascially we are trying uprobe... I getting following error

Failed to execute script Compiler error on line 42, column 38: 'pxtrace' object has no attribute 'uprobe'.

So was checking if there support for uprobe, if yes, can someone tell me what the problem could be the above pxl script

[ddelnano]

My earlier comment about the lack of uprobe support was prior to hearing that James mentioned there is undocumented support.

@mighty1231 I believe there is the ability to a shared object. The pxtrace.UpsertTracepoint's target argument can be a pxtrace.SharedObject object.

@josyulav there is no pxtrace.uprobe. I haven't been able to test this myself, but I believe what you are looking for is one of px.SharedObject or pxtrace.PodProcess in place of your pxtrace.uprobe.