search

google / perfetto

Performance instrumentation and tracing for Android, Linux and Chrome (read-only mirror of https://android.googlesource.com/platform/external/perfetto/)
https://www.perfetto.dev
Apache License 2.0
2.82k stars 352 forks source link

Questions about TraceProcessor #908

Closed wangkai0055 closed 1 week ago

wangkai0055 commented 1 week ago

1. When I use bin, can I load the format of trace.html? This will return an error. TraceProcessor(trace='trace.html', config=TraceProcessorConfig(bin_path='./trace_processor') 2. tp = TraceProcessor(addr='localhost:10000') qr_it = tp.query("SELECT ts, dur/1000000 as dur, name FROM slice")

socket.gaierror: [Errno 11001] getaddrinfo failed

LalitMaganti commented 1 week ago

When I use bin, can I load the format of trace.html?

Can you please provide what error it returns? There is no restriction on trace format. If it can be loaded in the Perfetto UI/in trace processor on the command line, it can also be loaded in the Python API.

socket.gaierror: [Errno 11001] getaddrinfo failed

Are you sure you have trace processor running on port 10000? Usually trace processor runs at port 9001 not 10000.

wangkai0055 commented 1 week ago

first question

File "AI.py", line 100, in process_file tp = TraceProcessor(trace=r"3.html", config=TraceProcessorConfig(bin_path='./trace_processor')) File "/home/wangkai/.local/lib/python3.8/site-packages/perfetto/trace_processor/api.py", line 260, in init self.http = self._create_tp_http(addr) File "/home/wangkai/.local/lib/python3.8/site-packages/perfetto/trace_processor/api.py", line 330, in _create_tp_http url, self.subprocess = load_shell(self.config.bin_path, File "/home/wangkai/.local/lib/python3.8/site-packages/perfetto/trace_processor/shell.py", line 65, in load_shell raise Exception( Exception: Trace processor failed to start. Try rerunning with verbose=True in TraceProcessorConfig for more detailed information and file a bug at https://goto.google.com/perfetto-bug or https://github.com/google/perfetto/issues if necessary.

LalitMaganti commented 1 week ago

The problem is not with opening on the HTML file but some network issue. The first time you run the trace_processor script, it tries to download the appropriate C++ binary. Seems like you don't have access to GCS which is why it's failing.

Instead of using the trace_processor script, you can download the appropriate zip file from https://github.com/google/perfetto/releases/tag/v48.0, extract the trace_processor_shell binary and point the Python API at that. That will bypass the network download.

wangkai0055 commented 1 week ago

Is the Python API modified like this?

tp = TraceProcessor(trace=r"3.html", config=TraceProcessorConfig(bin_path='./trace_processor_shell')) File "AI.py", line 100, in process_file tp = TraceProcessor(trace=r"3.html", config=TraceProcessorConfig(bin_path='./trace_processor_shell')) File "/home/wangkai/.local/lib/python3.8/site-packages/perfetto/trace_processor/api.py", line 260, in init self.http = self._create_tp_http(addr) File "/home/wangkai/.local/lib/python3.8/site-packages/perfetto/trace_processor/api.py", line 330, in _create_tp_http url, self.subprocess = load_shell(self.config.bin_path, File "/home/wangkai/.local/lib/python3.8/site-packages/perfetto/trace_processor/shell.py", line 65, in load_shell raise Exception( Exception: Trace processor failed to start. Try rerunning with verbose=True in TraceProcessorConfig for more detailed information and file a bug at https://goto.google.com/perfetto-bug or https://github.com/google/perfetto/issues if necessary

LalitMaganti commented 1 week ago

Instead of using the Python API, can you try just running ./trace_processor_shell 3.html on the command line?

wangkai0055 commented 1 week ago

I can do this on Windows, but Linux will report an error. wangkai@dg-72112803d01:/mnt/e/OPEN$ ./trace_processor_shell 3.html ./trace_processor_shell: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by ./trace_processor_shell) ./trace_processor_shell: /lib/x86_64-linux-gnu/libc.so.6: versionGLIBC_2.34' not found (required by ./trace_processor_shell)

Also, I need to query in Python, how to get the results, just like panda. Is there any API?

LalitMaganti commented 1 week ago

I can do this on Windows, but Linux

Seems like your version of Linux is too old. glibc 2.33 was released in 2021 (https://sourceware.org/pipermail/libc-alpha/2021-February/122207.html) so your Linux version has some version before this.

Also, I need to query in Python, how to get the results, just like panda. Is there any API?

The Python API is designed for this but under the hood it does the ./trace_processor_shell command. The glibc failure is also the root cause of the Python API failing.

LalitMaganti commented 1 week ago

If you want to work around this, you can build trace_processor_shell from source (see instructions at https://perfetto.dev/docs/contributing/getting-started) and then if you copy out/linux_clang_release/trace_processor_shell build artifact to your Python script, tp = TraceProcessor(trace=r"3.html", config=TraceProcessorConfig(bin_path='./trace_processor_shell')) should work again.

wangkai0055 commented 1 week ago

If I want to build a Python API for Windows, do I have to use the Windows system? Or can Linux also compile the Windows API?

LalitMaganti commented 1 week ago

If I want to build a Python API for Windows, do I have to use the Windows system?

On Windows, if ./trace_processor_shell 3.html works out of the box, then the Python API should also just work. You would not need to recompile anything.

If it doesn't work, please provide the output of tp = TraceProcessor(trace=r"3.html", config=TraceProcessorConfig(bin_path='trace_processor_shell.exe'), verbose=True) (after putting the shell exe in the correct place so the Python script will find it).

wangkai0055 commented 1 week ago

请输入文件或文件夹路径: hello.html SyntaxError: Non-UTF-8 code starting with '\x98' in file E:\OPEN\trace_processor_shell on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details Traceback (most recent call last): File "E:\OPEN\AI.py", line 137, in traverse_and_process(input_path) File "E:\OPEN\AI.py", line 127, in traverse_and_process process_file(path) File "E:\OPEN\AI.py", line 100, in process_file tp = TraceProcessor(trace=r"hello.html", config=TraceProcessorConfig(bin_path='./trace_processor_shell',verbose=True)) File "C:\Users\72112803\AppData\Local\Programs\Python\Python310\lib\site-packages\perfetto\trace_processor\api.py", line 260, in init self.http = self._create_tp_http(addr) File "C:\Users\72112803\AppData\Local\Programs\Python\Python310\lib\site-packages\perfetto\trace_processor\api.py", line 330, in _create_tp_http url, self.subprocess = load_shell(self.config.bin_path, File "C:\Users\72112803\AppData\Local\Programs\Python\Python310\lib\site-packages\perfetto\trace_processor\shell.py", line 65, in load_shell raise Exception( Exception: Trace processor failed to start. Try rerunning with verbose=True in TraceProcessorConfig for more detailed information and file a bug at https://goto.google.com/perfetto-bug or https://github.com/google/perfetto/issues if necessary.

LalitMaganti commented 1 week ago

Please read the command I gave in the previous message carefully: what you ran does not match the Python I provided.

wangkai0055 commented 1 week ago

请输入文件或文件夹路径: 3.html [830.618] httpd.cc:102 [HTTP] Starting RPC server on localhost:1719 [830.620] httpd.cc:107 [HTTP] This server can be used by reloading https://ui.perfetto.dev and clicking on YES on the "Trace Processor native acceleration" dialog or through the Python API (see https://perfetto.dev/docs/analysis/trace-processor#python-api). Traceback (most recent call last): File "E:\OPEN\AI.py", line 137, in traverse_and_process(input_path) File "E:\OPEN\AI.py", line 127, in traverse_and_process process_file(path) File "E:\OPEN\AI.py", line 100, in process_file tp = TraceProcessor(trace=r"3.html", config=TraceProcessorConfig(bin_path='trace_processor_shell.exe',verbose=True)) File "C:\Users\72112803\AppData\Local\Programs\Python\Python310\lib\site-packages\perfetto\trace_processor\api.py", line 260, in init self.http = self._create_tp_http(addr) File "C:\Users\72112803\AppData\Local\Programs\Python\Python310\lib\site-packages\perfetto\trace_processor\api.py", line 330, in _create_tp_http url, self.subprocess = load_shell(self.config.bin_path, File "C:\Users\72112803\AppData\Local\Programs\Python\Python310\lib\site-packages\perfetto\trace_processor\shell.py", line 65, in load_shell raise Exception( Exception: Trace processor failed to start. Try rerunning with verbose=True in TraceProcessorConfig for more detailed information and file a bug at https://goto.google.com/perfetto-bug or https://github.com/google/perfetto/issues if necessary.

LalitMaganti commented 1 week ago 
[830.618] httpd.cc:102 [HTTP] Starting RPC server on localhost:1719
[830.620] httpd.cc:107 [HTTP] This server can be used by reloading https://ui.perfetto.dev and clicking on YES on the "Trace Processor native acceleration" dialog or through the Python API (see https://perfetto.dev/docs/analysis/trace-processor#python-api).

These log lines indicate that trace processor was able to start successfully meaning there is some communication issue between Python and the trace processor subprocess. There might be something (maybe a firewall or a anti-virus software) blocking the communication.

It's very difficult without access to a similar setup to debug this issue I'm afraid. But from the Perfetto side, there does not appear to be anything wrong per se.

LalitMaganti commented 1 week ago

Note that very recently (in https://github.com/google/perfetto/issues/895), I verified that the Python API works correctly on Windows by doing a fresh install.

wangkai0055 commented 1 week ago

Is there any other way to solve it? Can we build the UI locally without using the network? Most companies have firewalls set up to prevent information leakage.

LalitMaganti commented 1 week ago

The use of HTTP on localhost is fundamental to how the Python API works. There are no workarounds I'm afraid, this is just how it works.

Note that your firewall may be overly aggresive here: we are only binding to the local interface which is only accessible on your local machine nowhere else. We don't expose the port to the LAN or WAN interface.

LalitMaganti commented 1 week ago

BTW when you say:

Can we build the UI locally without using the network?

Not sure how the UI is relevant here: this is a Python API question only. The Perfetto UI (ui.perfetto.dev) does not open any ports nor does it require internet access (once you access it for the first time).

Is there any other way to solve it?

The only other thing you could do is not use the Python API but instead use trace_processor_shell directly and output CSV files which you can then import into Pandas. This is obviously nowhere near as good as the integrated experience you get with using the Python API but may work for your needs potentially.