search

google / perfetto

Performance instrumentation and tracing for Android, Linux and Chrome (read-only mirror of https://android.googlesource.com/platform/external/perfetto/)
https://www.perfetto.dev
Apache License 2.0
2.93k stars 362 forks source link

heap_profile: wide `unknown` instances in callstack for emulator #112

Closed nickaein closed 3 years ago

nickaein commented 3 years ago

I'm using the following configuration to profile the release build of our app on Android Virtual Device (Pixel 4 API 29):

PERFETTO_BINARY_PATH=~/app/build/intermediates/cmake/release/obj/x86
./heap_profile -n app.name -i 65536 -c 10000 --shmem-size 65536

The generated symbolized-trace (around ~71KB), has a flamegraph with wide instances of unknown. A readable callstack is crammed into the right side of chart, but I'm not sure whether that's the expected callstack or some noise.

image

However, using the same configuration with similar profiling scenario, heap_profile works as expected on an actual device (Samsung A51 running Android 11):

PERFETTO_BINARY_PATH=~/app/build/intermediates/cmake/release/obj/arm64-v8a
./heap_profile -n app.name -i 65536 -c 10000 --shmem-size 65536

The symbolized-trace (around ~2MB which is much larger than previous one) is as expected:

image

primiano commented 3 years ago

@fmayer this might be something ABI-related. Not sure what's the state of unwinding on x86 and x86_64 devices.

nickaein commented 3 years ago

Just as reference, I've included the output of readelf for x86 and arm64-v8a libraries. Both have .eh_frame and .eh_frame_hdr seems to be enough based on the documentation.

I'm also getting a ton of local_symbolizer.cc:457 Could not find Build ID: warnings, but they are related to system/android libraries which is fine if their callstack doesn't unwind.

readelf output for x86 library (for emulator):

$ readelf -S ~/app/build/intermediates/cmake/release/obj/x86/libmapbox-gl.so 
There are 39 section headers, starting at offset 0x11149990:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .note.android.ide NOTE            00000134 000134 000098 00   A  0   0  2
  [ 2] .note.gnu.build-i NOTE            000001cc 0001cc 000024 00   A  0   0  4
  [ 3] .dynsym           DYNSYM          000001f0 0001f0 009ac0 10   A  4   1  4
  [ 4] .dynstr           STRTAB          00009cb0 009cb0 0199d0 00   A  0   0  1
  [ 5] .gnu.hash         GNU_HASH        00023680 023680 00495c 04   A  3   0  4
  [ 6] .hash             HASH            00027fdc 027fdc 0046cc 04   A  3   0  4
  [ 7] .gnu.version      VERSYM          0002c6a8 02c6a8 001358 02   A  3   0  2
  [ 8] .gnu.version_d    VERDEF          0002da00 02da00 00001c 00   A  4   1  4
  [ 9] .gnu.version_r    VERNEED         0002da1c 02da1c 000060 00   A  4   3  4
  [10] .rel.dyn          REL             0002da7c 02da7c 018998 08   A  3   0  4
  [11] .rel.plt          REL             00046414 046414 002200 08  AI  3  12  4
  [12] .plt              PROGBITS        00048620 048620 004410 04  AX  0   0 16
  [13] .text             PROGBITS        0004ca30 04ca30 10fea00 00  AX  0   0 16
  [14] .rodata           PROGBITS        0114b430 114b430 0e8694 00   A  0   0 16
  [15] .gcc_except_table PROGBITS        01233ac4 1233ac4 0a09ec 00   A  0   0  4
  [16] .eh_frame         PROGBITS        012d44b0 12d44b0 44f0a0 00   A  0   0  4
  [17] .eh_frame_hdr     PROGBITS        01723550 1723550 0ee9ec 00   A  0   0  4
  [18] .data.rel.ro      PROGBITS        018139fc 18129fc 00dfac 00  WA  0   0  4
  [19] .fini_array       FINI_ARRAY      018219a8 18209a8 000008 00  WA  0   0  4
  [20] .init_array       INIT_ARRAY      018219b0 18209b0 000088 00  WA  0   0  4
  [21] .dynamic          DYNAMIC         01821a38 1820a38 000148 08  WA  4   0  4
  [22] .got              PROGBITS        01821b80 1820b80 000374 00  WA  0   0  4
  [23] .got.plt          PROGBITS        01821ef4 1820ef4 00110c 00  WA  0   0  4
  [24] .data             PROGBITS        01823000 1822000 000e48 00  WA  0   0  8
  [25] .bss              NOBITS          01823e80 1822e80 006b60 00  WA  0   0 64
  [26] .comment          PROGBITS        00000000 1822e48 0000dc 01  MS  0   0  1
  [27] .debug_str        PROGBITS        00000000 1822f24 3bf1bc1 01  MS  0   0  1
  [28] .debug_abbrev     PROGBITS        00000000 5414ae5 0ef5bd 00      0   0  1
  [29] .debug_info       PROGBITS        00000000 55040a2 6091a60 00      0   0  1
  [30] .debug_macinfo    PROGBITS        00000000 b595b02 0001da 00      0   0  1
  [31] .debug_line       PROGBITS        00000000 b595cdc 12df1de 00      0   0  1
  [32] .debug_ranges     PROGBITS        00000000 c874eba 2c2de8 00      0   0  1
  [33] .debug_loc        PROGBITS        00000000 cb37ca2 0c3a8b 00      0   0  1
  [34] .debug_aranges    PROGBITS        00000000 cbfb72d 0000e0 00      0   0  1
  [35] .note.gnu.gold-ve NOTE            00000000 cbfb810 00001c 00      0   0  4
  [36] .symtab           SYMTAB          00000000 cbfb82c 2ea2600 10     37 3053749  4
  [37] .strtab           STRTAB          00000000 fa9de2c 16ab9c3 00      0   0  1
  [38] .shstrtab         STRTAB          00000000 111497ef 0001a0 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

readelf output for arm64-v8a (for A51 device):

$ readelf -S ~/app/build/intermediates/cmake/release/obj/arm64-v8a/libmapbox-gl.so 
There are 36 section headers, starting at offset 0xeac3af8:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.gnu.build-i NOTE             0000000000000200  00000200
       0000000000000024  0000000000000000   A       0     0     4
  [ 2] .hash             HASH             0000000000000228  00000228
       000000000000475c  0000000000000004   A       4     0     8
  [ 3] .gnu.hash         GNU_HASH         0000000000004988  00004988
       0000000000004968  0000000000000000   A       4     0     8
  [ 4] .dynsym           DYNSYM           00000000000092f0  000092f0
       000000000000eb80  0000000000000018   A       5     3     8
  [ 5] .dynstr           STRTAB           0000000000017e70  00017e70
       0000000000019a7d  0000000000000000   A       0     0     1
  [ 6] .gnu.version      VERSYM           00000000000318ee  000318ee
       00000000000013a0  0000000000000002   A       4     0     2
  [ 7] .gnu.version_r    VERNEED          0000000000032c90  00032c90
       0000000000000060  0000000000000000   A       5     3     8
  [ 8] .rela.dyn         RELA             0000000000032cf0  00032cf0
       000000000004ae20  0000000000000018   A       4     0     8
  [ 9] .rela.plt         RELA             000000000007db10  0007db10
       0000000000003f90  0000000000000018  AI       4    21     8
  [10] .plt              PROGBITS         0000000000081aa0  00081aa0
       0000000000002a80  0000000000000010  AX       0     0     16
  [11] .text             PROGBITS         0000000000084520  00084520
       0000000000c5b100  0000000000000000  AX       0     0     4
  [12] .rodata           PROGBITS         0000000000cdf620  00cdf620
       00000000000e6fed  0000000000000000   A       0     0     16
  [13] .eh_frame_hdr     PROGBITS         0000000000dc6610  00dc6610
       00000000000eef04  0000000000000000   A       0     0     4
  [14] .eh_frame         PROGBITS         0000000000eb5518  00eb5518
       000000000039b138  0000000000000000   A       0     0     8
  [15] .gcc_except_table PROGBITS         0000000001250650  01250650
       00000000000ab8c8  0000000000000000   A       0     0     4
  [16] .note.android.ide NOTE             00000000012fbf18  012fbf18
       0000000000000098  0000000000000000   A       0     0     4
  [17] .init_array       INIT_ARRAY       00000000012fd098  012fc098
       0000000000000110  0000000000000008  WA       0     0     8
  [18] .fini_array       FINI_ARRAY       00000000012fd1a8  012fc1a8
       0000000000000010  0000000000000008  WA       0     0     8
  [19] .data.rel.ro      PROGBITS         00000000012fd1b8  012fc1b8
       000000000001b948  0000000000000000  WA       0     0     8
  [20] .dynamic          DYNAMIC          0000000001318b00  01317b00
       0000000000000260  0000000000000010  WA       5     0     8
  [21] .got              PROGBITS         0000000001318d60  01317d60
       00000000000022a0  0000000000000008  WA       0     0     8
  [22] .data             PROGBITS         000000000131b000  0131a000
       0000000000001bb0  0000000000000000  WA       0     0     8
  [23] .bss              NOBITS           000000000131cbb0  0131bbb0
       0000000000009cf0  0000000000000000  WA       0     0     16
  [24] .comment          PROGBITS         0000000000000000  0131bbb0
       00000000000000db  0000000000000001  MS       0     0     1
  [25] .debug_aranges    PROGBITS         0000000000000000  0131bc8b
       0000000000000300  0000000000000000           0     0     1
  [26] .debug_info       PROGBITS         0000000000000000  0131bf8b
       0000000006226d4b  0000000000000000           0     0     1
  [27] .debug_abbrev     PROGBITS         0000000000000000  07542cd6
       00000000000f335e  0000000000000000           0     0     1
  [28] .debug_line       PROGBITS         0000000000000000  07636034
       000000000147912b  0000000000000000           0     0     1
  [29] .debug_str        PROGBITS         0000000000000000  08aaf15f
       0000000003b210ad  0000000000000001  MS       0     0     1
  [30] .debug_loc        PROGBITS         0000000000000000  0c5d020c
       000000000019bc6b  0000000000000000           0     0     1
  [31] .debug_macinfo    PROGBITS         0000000000000000  0c76be77
       00000000000001cb  0000000000000000           0     0     1
  [32] .debug_ranges     PROGBITS         0000000000000000  0c76c042
       00000000005c0310  0000000000000000           0     0     1
  [33] .shstrtab         STRTAB           0000000000000000  0eac3980
       0000000000000173  0000000000000000           0     0     1
  [34] .symtab           SYMTAB           0000000000000000  0cd2c358
       00000000006d5528  0000000000000018          35   296042     8
  [35] .strtab           STRTAB           0000000000000000  0d401880
       00000000016c2100  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

heap_profile output on emulator:

Profiling active. Press Ctrl+C to terminate.
You may disconnect your device.

^CWaiting for profiler shutdown...
[253.645] local_symbolizer.cc:457 Could not find Build ID: 44af978b95c1b5bd57561a6f71f12187 (file /apex/com.android.conscrypt/lib/libc++.so).
[253.645] local_symbolizer.cc:457 Could not find Build ID: 1532d9cb7b20bba36dd34cb98d4dcd1f (file /apex/com.android.conscrypt/lib/libjavacrypto.so).
[253.645] local_symbolizer.cc:457 Could not find Build ID: 76290498408016ad14f4b98c3ab6c65c (file /apex/com.android.runtime/lib/bionic/libc.so).
[253.645] local_symbolizer.cc:457 Could not find Build ID: 76290498408016ad14f4b98c3ab6c65c (file /apex/com.android.runtime/lib/bionic/libc.so!libc.so).
[253.645] local_symbolizer.cc:457 Could not find Build ID: 895645e5113da057f27d9b2ec11eb3bf (file /apex/com.android.runtime/lib/libart.so).
[253.645] local_symbolizer.cc:457 Could not find Build ID: a232e2b15597ff6f7f8602680097e25a (file /apex/com.android.runtime/lib/libicuuc.so).
[253.737] local_symbolizer.cc:457 Could not find Build ID: b7a60bc7d078521421fd5a8d201915ae (file /system/bin/app_process32).
[253.737] local_symbolizer.cc:457 Could not find Build ID: ff6ec03dd8445d20788424c92ba8ea28ad0f54f4 (file /system/framework/x86/boot-framework.oat).
[253.737] local_symbolizer.cc:457 Could not find Build ID: 9427bc5d1e5194b92b7a0b4b481c969749980c34 (file /system/framework/x86/boot-okhttp.oat).
[253.737] local_symbolizer.cc:457 Could not find Build ID: 7913dbaef2e8d9971cb7619ef0d566987f8326a7 (file /system/framework/x86/boot.oat).
[253.737] local_symbolizer.cc:457 Could not find Build ID: aff062dd6d243aed75e0dfffaa89bce0 (file /system/lib/heapprofd_client.so).
[253.737] local_symbolizer.cc:457 Could not find Build ID: 6ceb9761bceb97a18c92f8a4b7072247 (file /system/lib/libandroid_runtime.so).
[253.737] local_symbolizer.cc:457 Could not find Build ID: 309011489c2c02a6361a247e89e8fdd4 (file /system/lib/libinput.so).
[253.737] local_symbolizer.cc:457 Could not find Build ID: 288ba3aff5b46dbd7e74be954af88b83 (file /system/lib/libutils.so).
[253.746] local_symbolizer.cc:457 Could not find Build ID: 44af978b95c1b5bd57561a6f71f12187 (file /apex/com.android.conscrypt/lib/libc++.so).
[253.746] local_symbolizer.cc:457 Could not find Build ID: 1532d9cb7b20bba36dd34cb98d4dcd1f (file /apex/com.android.conscrypt/lib/libjavacrypto.so).
[253.746] local_symbolizer.cc:457 Could not find Build ID: 76290498408016ad14f4b98c3ab6c65c (file /apex/com.android.runtime/lib/bionic/libc.so).
[253.746] local_symbolizer.cc:457 Could not find Build ID: 76290498408016ad14f4b98c3ab6c65c (file /apex/com.android.runtime/lib/bionic/libc.so!libc.so).
[253.746] local_symbolizer.cc:457 Could not find Build ID: 895645e5113da057f27d9b2ec11eb3bf (file /apex/com.android.runtime/lib/libart.so).
[253.746] local_symbolizer.cc:457 Could not find Build ID: a232e2b15597ff6f7f8602680097e25a (file /apex/com.android.runtime/lib/libicuuc.so).
[253.784] local_symbolizer.cc:457 Could not find Build ID: b7a60bc7d078521421fd5a8d201915ae (file /system/bin/app_process32).
[253.784] local_symbolizer.cc:457 Could not find Build ID: ff6ec03dd8445d20788424c92ba8ea28ad0f54f4 (file /system/framework/x86/boot-framework.oat).
[253.784] local_symbolizer.cc:457 Could not find Build ID: 9427bc5d1e5194b92b7a0b4b481c969749980c34 (file /system/framework/x86/boot-okhttp.oat).
[253.784] local_symbolizer.cc:457 Could not find Build ID: 7913dbaef2e8d9971cb7619ef0d566987f8326a7 (file /system/framework/x86/boot.oat).
[253.784] local_symbolizer.cc:457 Could not find Build ID: aff062dd6d243aed75e0dfffaa89bce0 (file /system/lib/heapprofd_client.so).
[253.784] local_symbolizer.cc:457 Could not find Build ID: 6ceb9761bceb97a18c92f8a4b7072247 (file /system/lib/libandroid_runtime.so).
[253.784] local_symbolizer.cc:457 Could not find Build ID: 309011489c2c02a6361a247e89e8fdd4 (file /system/lib/libinput.so).
[253.784] local_symbolizer.cc:457 Could not find Build ID: 288ba3aff5b46dbd7e74be954af88b83 (file /system/lib/libutils.so).
Wrote profiles to /tmp/8d27a6 (symlink /tmp/heap_profile-latest)
These can be viewed using pprof. Googlers: head to pprof/ and upload them.

heap_profile output on A51 device:

If you are experiencing problems, please see the known issues for your release: https://perfetto.dev/docs/data-sources/native-heap-profiler#known-issues-android11.
Profiling active. Press Ctrl+C to terminate.
You may disconnect your device.

^CWaiting for profiler shutdown...
[376.070] local_symbolizer.cc:457 Could not find Build ID: 39d018698166872c36fbf98696dcd533655f6577 (file /apex/com.android.art/javalib/arm64/boot-core-icu4j.oat).
[376.070] local_symbolizer.cc:457 Could not find Build ID: ac2e1e88df26e560d045a5d0ae262bcb775bfd5d (file /apex/com.android.art/javalib/arm64/boot-core-libart.oat).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 29f4d6635b922ec5a4805e0ee459f65b7d32ec93 (file /apex/com.android.art/javalib/arm64/boot-okhttp.oat).
[376.070] local_symbolizer.cc:457 Could not find Build ID: f8c286bf3b204e4b73705f3f371a4596c2d087b2 (file /apex/com.android.art/javalib/arm64/boot.oat).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 5809e26a3f785e363e7b554d1abc5c75 (file /apex/com.android.art/lib64/libart-compiler.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: c0bd7f6d76014552240c0a4744b9368d (file /apex/com.android.art/lib64/libart.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 00fc0962a5f6f85a55f15e85e02513b3 (file /apex/com.android.art/lib64/libicu_jni.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: bf9001e8c044b4f34c688fd07878e060 (file /apex/com.android.art/lib64/libicui18n.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: bc9b5b11ffb031d9c0ac287287b99536 (file /apex/com.android.art/lib64/libicuuc.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 584b4f0d0867de3eeeab5c86f341ebc0 (file /apex/com.android.art/lib64/libjavacore.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 2fd4518e93785963497875e583dbbdea (file /apex/com.android.art/lib64/libprofile.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 434b6e895068cceee3af11deefaf4505 (file /apex/com.android.art/lib64/libvixl.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /apex/com.android.conscrypt/lib64/libcrypto.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /apex/com.android.conscrypt/lib64/libcrypto.so!libcrypto.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 774cec9038833dd1f6f88618ec1d59b0 (file /apex/com.android.conscrypt/lib64/libjavacrypto.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: e4f77e81faeacae32860a065929b0c52 (file /apex/com.android.conscrypt/lib64/libssl.so).
[376.070] local_symbolizer.cc:457 Could not find Build ID: 4c0c87e964ad0db6a7f358c8ac661d07 (file /apex/com.android.runtime/lib64/bionic/libc.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 2ad8463550327212cdc6054116089674 (file /system/bin/app_process64).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 9dad242773b9949e67c2a06f1b4a7faa47bf971e (file /system/framework/arm64/boot-framework.oat).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 544e89f0a34970c856b972b35256911f (file /system/lib64/libEGL.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: b06095de25bda69601ad25291b6017d5 (file /system/lib64/libandroid.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 5b92b6f8ab5ad39f680898f9fe8fbd5d (file /system/lib64/libandroid_runtime.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: bd562ced0591d96d72aadab3cd297025 (file /system/lib64/libbinder.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: da087bc6aeeb87276326ccdef0342b02 (file /system/lib64/libc++.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /system/lib64/libcrypto.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /system/lib64/libcrypto.so!libcrypto.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 1f5d49b393c84f37de5b20e99ffaa73d (file /system/lib64/libgui.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: fc533bc7ac254d726057340ba6ce613f (file /system/lib64/libhwui.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 4806c7fb4359014272629b4a9b8885d0 (file /system/lib64/libinput.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 4e69b93bf70ed592f0029dbd1097529e (file /system/lib64/libutils.so).
[376.287] local_symbolizer.cc:457 Could not find Build ID: 60623782311f3e1ec5dabccf267e821d (file /vendor/lib64/egl/libGLES_mali.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 39d018698166872c36fbf98696dcd533655f6577 (file /apex/com.android.art/javalib/arm64/boot-core-icu4j.oat).
[376.307] local_symbolizer.cc:457 Could not find Build ID: ac2e1e88df26e560d045a5d0ae262bcb775bfd5d (file /apex/com.android.art/javalib/arm64/boot-core-libart.oat).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 29f4d6635b922ec5a4805e0ee459f65b7d32ec93 (file /apex/com.android.art/javalib/arm64/boot-okhttp.oat).
[376.307] local_symbolizer.cc:457 Could not find Build ID: f8c286bf3b204e4b73705f3f371a4596c2d087b2 (file /apex/com.android.art/javalib/arm64/boot.oat).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 5809e26a3f785e363e7b554d1abc5c75 (file /apex/com.android.art/lib64/libart-compiler.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: c0bd7f6d76014552240c0a4744b9368d (file /apex/com.android.art/lib64/libart.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 00fc0962a5f6f85a55f15e85e02513b3 (file /apex/com.android.art/lib64/libicu_jni.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: bf9001e8c044b4f34c688fd07878e060 (file /apex/com.android.art/lib64/libicui18n.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: bc9b5b11ffb031d9c0ac287287b99536 (file /apex/com.android.art/lib64/libicuuc.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 584b4f0d0867de3eeeab5c86f341ebc0 (file /apex/com.android.art/lib64/libjavacore.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 2fd4518e93785963497875e583dbbdea (file /apex/com.android.art/lib64/libprofile.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 434b6e895068cceee3af11deefaf4505 (file /apex/com.android.art/lib64/libvixl.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /apex/com.android.conscrypt/lib64/libcrypto.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /apex/com.android.conscrypt/lib64/libcrypto.so!libcrypto.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 774cec9038833dd1f6f88618ec1d59b0 (file /apex/com.android.conscrypt/lib64/libjavacrypto.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: e4f77e81faeacae32860a065929b0c52 (file /apex/com.android.conscrypt/lib64/libssl.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 4c0c87e964ad0db6a7f358c8ac661d07 (file /apex/com.android.runtime/lib64/bionic/libc.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 2ad8463550327212cdc6054116089674 (file /system/bin/app_process64).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 9dad242773b9949e67c2a06f1b4a7faa47bf971e (file /system/framework/arm64/boot-framework.oat).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 544e89f0a34970c856b972b35256911f (file /system/lib64/libEGL.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: b06095de25bda69601ad25291b6017d5 (file /system/lib64/libandroid.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 5b92b6f8ab5ad39f680898f9fe8fbd5d (file /system/lib64/libandroid_runtime.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: bd562ced0591d96d72aadab3cd297025 (file /system/lib64/libbinder.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: da087bc6aeeb87276326ccdef0342b02 (file /system/lib64/libc++.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /system/lib64/libcrypto.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 0eea544a320e826d4fef4540030dfdb1 (file /system/lib64/libcrypto.so!libcrypto.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 1f5d49b393c84f37de5b20e99ffaa73d (file /system/lib64/libgui.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: fc533bc7ac254d726057340ba6ce613f (file /system/lib64/libhwui.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 4806c7fb4359014272629b4a9b8885d0 (file /system/lib64/libinput.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 4e69b93bf70ed592f0029dbd1097529e (file /system/lib64/libutils.so).
[376.307] local_symbolizer.cc:457 Could not find Build ID: 60623782311f3e1ec5dabccf267e821d (file /vendor/lib64/egl/libGLES_mali.so).
Wrote profiles to /tmp/9cf455 (symlink /tmp/heap_profile-latest)
These can be viewed using pprof. Googlers: head to pprof/ and upload them.
fmayer commented 3 years ago

In case your emulator is x86_64 and your app is x86, could you try fetching the newest version of Perfetto? I resolved a cross-ABI issue yesterday.

nickaein commented 3 years ago

I believe the emulator is x86. I don't have any preference between x86 and x86_64 architectures tough, and just followed the default AVD choices to create emulator. I can switch x86_64 if it is recommended one. The only tweak I have done to emulator is increasing the emulator hw.ramSize to 8536 MBs.

I git pulled the latest version (b0d9de4c263359c154e3ca31af9d40055f9840ad) and I'm still getting the same result with tools/heap_profile.

I noticed that the trace_to_text binary (located at /tmp/trace_to_text-linux-0df4f571099135a29659caf35fd5e343ef59bdc3) seems to be unchanged, but the changes you have done (this commit? a4edfc0db10b623d6fdf3281b358bc131bade994) seems to be in C++ codes. Is that fine or I have to rebuild some C++ binaries?

My ~/.android/avd/Pixel_4_API_29.avd/config.ini:

Name: Pixel_4_API_29
CPU/ABI: Google Play Intel Atom (x86)
Path: /home/isaac/.android/avd/Pixel_4_API_29.avd
Target: google_apis_playstore [Google Play] (API level 29)
Skin: pixel_4
SD Card: 512 MB
fastboot.chosenSnapshotFile: 
runtime.network.speed: full
hw.accelerometer: yes
hw.device.name: pixel_4
hw.lcd.width: 1080
hw.initialOrientation: Portrait
image.androidVersion.api: 29
tag.id: google_apis_playstore
hw.mainKeys: no
hw.camera.front: emulated
avd.ini.displayname: Pixel 4 API 29
hw.gpu.mode: auto
hw.ramSize: 8536
PlayStore.enabled: true
fastboot.forceColdBoot: no
hw.cpu.ncore: 4
hw.keyboard: yes
hw.sensors.proximity: yes
hw.dPad: no
hw.lcd.height: 2280
vm.heapSize: 256
skin.dynamic: yes
hw.device.manufacturer: Google
hw.gps: yes
hw.audioInput: yes
image.sysdir.1: system-images/android-29/google_apis_playstore/x86/
showDeviceFrame: yes
hw.camera.back: virtualscene
AvdId: Pixel_4_API_29
hw.lcd.density: 440
hw.arc: false
hw.device.hash2: MD5:6b5943207fe196d842659d2e43022e20
fastboot.forceChosenSnapshotBoot: no
fastboot.forceFastBoot: yes
hw.trackBall: no
hw.battery: yes
hw.sdCard: yes
tag.display: Google Play
runtime.network.latency: none
disk.dataPartition.size: 6G
hw.sensors.orientation: yes
avd.ini.encoding: UTF-8
hw.gpu.enabled: yes
fmayer commented 3 years ago

Ah! I thought you were building the emulator yourself, my change was to Android itself, not the Perfetto tooling.

It seems like your emulator is running Android 10 (SDK 29), on which we unfortunately do not support profiling on X86 (see https://perfetto.dev/docs/data-sources/native-heap-profiler#known-issues-android10).

fmayer commented 3 years ago

That means trying x86_64 (and building your app for that as well) should resolve the issue. Let me know how that works if you get a chance to try.

nickaein commented 3 years ago

Whoops! Sorry that I missed the documentation.

I created another device in AVD based on Android 10 x86_64 image but I'm still having the same issue (unknown instances in flamegraph), whether I set PERFETTO_BINARY_PATH to x86_64 or x86 library path.

Should I try with Android 11 (R - API Level 30) x86_64 image?

config.ini of the new emulator:

$ cat ~/.android/avd/Pixel_4_API_29_x64.avd/config.ini
AvdId = Pixel_4_API_29_x64
PlayStore.enabled = true
abi.type = x86_64
avd.ini.displayname = Pixel 4 API 29 x64
avd.ini.encoding = UTF-8
disk.dataPartition.size = 6442450944
fastboot.chosenSnapshotFile = 
fastboot.forceChosenSnapshotBoot = no
fastboot.forceColdBoot = no
fastboot.forceFastBoot = yes
hw.accelerometer = yes
hw.arc = false
hw.audioInput = yes
hw.battery = yes
hw.camera.back = virtualscene
hw.camera.front = emulated
hw.cpu.arch = x86_64
hw.cpu.ncore = 4
hw.dPad = no
hw.device.hash2 = MD5:6b5943207fe196d842659d2e43022e20
hw.device.manufacturer = Google
hw.device.name = pixel_4
hw.gps = yes
hw.gpu.enabled = yes
hw.gpu.mode = auto
hw.initialOrientation = Portrait
hw.keyboard = yes
hw.lcd.density = 440
hw.lcd.height = 2280
hw.lcd.width = 1080
hw.mainKeys = no
hw.ramSize = 1536
hw.sdCard = yes
hw.sensors.orientation = yes
hw.sensors.proximity = yes
hw.trackBall = no
image.sysdir.1 = system-images/android-29/google_apis_playstore/x86_64/
runtime.network.latency = none
runtime.network.speed = full
sdcard.size = 512M
showDeviceFrame = yes
skin.dynamic = yes
skin.name = pixel_4
skin.path = /home/isaac/Android/Sdk/skins/pixel_4
tag.display = Google Play
tag.id = google_apis_playstore
vm.heapSize = 256
fmayer commented 3 years ago

Just confirming that you also built your target app as x86_64?

nickaein commented 3 years ago

AFAIK, yes. This is our setup:

The C++ library (with a java wrapper) is built in another project for all platforms and its stripped binaries are packaged and uploaded to an maven artifactory as a standalone package. The main android application uses this library as dependency and is "isolated" from native codes. I assume the Android Studio uploads the correct version of app (x86_64) when deploying and running app on the x86_64 emulator. Is this a valid assumption?

For heap_profile, I use one of the following directories as PERFETTO_BINARY_PATH based on the targeting platform, to provide intermediate "unstripped" binary of the library:

$ tree ./app/build/intermediates/cmake/release/obj/      ​
./app/build/intermediates/cmake/release/obj
├── arm64-v8a
│   └── libmapbox-gl.so
├── armeabi-v7a
│   └── libmapbox-gl.so
├── x86
│   └── libmapbox-gl.so
└── x86_64
   ​└── libmapbox-gl.so
nickaein commented 3 years ago

I tested on emulator with image Android 11 (x86_64) and the symbolizer seems to be working as expected! So in summary we have these cases:

Case 1 (callstack unwinding works fine):

Case 2 (callstack unwinding fails):

Notes:

  1. Initially, I incorrectly mentioned that A51 device is running Android 10, which is not. Sorry about that. The A51 device has recently received Android 11 update.

  2. In Case 1, there are duplicate warnings from symbolizer about the missing system/android libraries (logs for emulator for Android 11 is included. Other logs can be found in https://github.com/google/perfetto/issues/112#issuecomment-800145660). However, in Case 2 there is no duplicate warnings. I'm not sure this can hint on the root cause of issue.

heap_profile output on emulator x86_46/Android 11:

If you are experiencing problems, please see the known issues for your release: https://perfetto.dev/docs/data-sources/native-heap-profiler#known-issues-android11.
Profiling active. Press Ctrl+C to terminate.
You may disconnect your device.

^CWaiting for profiler shutdown...
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot-core-icu4j.oat (Build ID: dd9df75e0d4565f38c62cc780ce5f3e31cac783f).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot-core-libart.oat (Build ID: 06c723e2ff44ead3e0ba45af7efe3dfad2b4a85d).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot-okhttp.oat (Build ID: f48c19bf2641e96166730694b0c96eb70841b3df).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot.oat (Build ID: 063a75624879be7d4274ea340748b04abbd86b8d).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libart-compiler.so (Build ID: 83e829d5d5d5b30f996fe2ca87ce6a9c).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libart.so (Build ID: 7fbaf2a1a3317bd634b00eb90e32291e).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libicu_jni.so (Build ID: 13537977ca5ceb77d180cbf174a9c682).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libicuuc.so (Build ID: 0c7df5516d30e684a88715858556e9ae).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libjavacore.so (Build ID: aef33c3aa96a413aab33b192eb3de27c).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libopenjdk.so (Build ID: ade89c69a2fee527f19802a9d10f4170).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.conscrypt/lib64/libcrypto.so (Build ID: 5cb769a4cb25f6c811ae2b43035e3227).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.conscrypt/lib64/libjavacrypto.so (Build ID: ba8642c9e08b0b870e8f1a1bda7a9ef8).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.conscrypt/lib64/libssl.so (Build ID: a5a3bc7f8e9d41cf86a435fa84aabcc6).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.runtime/lib64/bionic/libc.so (Build ID: 3707c39fc397eeaa328142d90b50a973).
[495.744] local_symbolizer.cc:481 Could not find /apex/com.android.vndk.v30/lib64/libc++.so (Build ID: 66195581cc855ea56bf2663a42e9115a).
[495.744] local_symbolizer.cc:481 Could not find /system/bin/app_process64 (Build ID: ce3cc65283f87cf0181925988a809dbd).
[495.744] local_symbolizer.cc:481 Could not find /system/framework/x86_64/boot-framework.oat (Build ID: a08fb51e45c8bbd96cf277b15edc84a7eb971431).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libEGL.so (Build ID: 2a2c36a4dd977d11859dfc585e6bf4e1).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libandroid_runtime.so (Build ID: e2556b95e6251c592f076d0623b50aa4).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libandroidfw.so (Build ID: bcb17989bbed4323a42fd500ab4741ab).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libbinder.so (Build ID: f55f2948551e7a9225c5653544eb5571).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libc++.so (Build ID: 5f6fc494386331f011fecfcc8642076c).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libft2.so (Build ID: e404befe9aa2fea418ee0245169a5dbd).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libgui.so (Build ID: 02d9f58fa70710bc6aab51a60a634760).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libhwui.so (Build ID: a588f7ea4e724e9e5d49bce097dee793).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libjpeg.so (Build ID: e746c56e0087900778f27708a9a5ac31).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libminikin.so (Build ID: 31c51bfcba5208f0e17b91bc0195a6a4).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libpng.so (Build ID: 83c7bc99efdcdfd0c7c1fcdb15f8cc89).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libsqlite.so (Build ID: ab3d62ca2c0cb74b4855ae47a6789ca8).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libutils.so (Build ID: 658bede56204d492a28682a718f6f501).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libz.so (Build ID: 727ab7cd735ce4ea52dd3dfa77081220).
[495.744] local_symbolizer.cc:481 Could not find /system/lib64/libziparchive.so (Build ID: 5a97b5974211715f15a4d39ef24ee523).
[495.744] local_symbolizer.cc:481 Could not find /vendor/lib64/egl/libEGL_emulation.so (Build ID: 1c910424fcdb1518e1cd3989660d173c).
[495.744] local_symbolizer.cc:481 Could not find /vendor/lib64/hw/android.hardware.graphics.mapper@3.0-impl-ranchu.so (Build ID: 320d65953c80987dedddf42c7f95067a).
[495.744] local_symbolizer.cc:481 Could not find /vendor/lib64/libGLESv2_enc.so (Build ID: 532afe147eca1b8b8e4dc916da282f18).
[495.744] local_symbolizer.cc:481 Could not find /vendor/lib64/libOpenglCodecCommon.so (Build ID: ca1a7f59d6c91b54f4450902354cce86).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot-core-icu4j.oat (Build ID: dd9df75e0d4565f38c62cc780ce5f3e31cac783f).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot-core-libart.oat (Build ID: 06c723e2ff44ead3e0ba45af7efe3dfad2b4a85d).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot-okhttp.oat (Build ID: f48c19bf2641e96166730694b0c96eb70841b3df).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/javalib/x86_64/boot.oat (Build ID: 063a75624879be7d4274ea340748b04abbd86b8d).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libart-compiler.so (Build ID: 83e829d5d5d5b30f996fe2ca87ce6a9c).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libart.so (Build ID: 7fbaf2a1a3317bd634b00eb90e32291e).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libicu_jni.so (Build ID: 13537977ca5ceb77d180cbf174a9c682).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libicuuc.so (Build ID: 0c7df5516d30e684a88715858556e9ae).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libjavacore.so (Build ID: aef33c3aa96a413aab33b192eb3de27c).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.art/lib64/libopenjdk.so (Build ID: ade89c69a2fee527f19802a9d10f4170).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.conscrypt/lib64/libcrypto.so (Build ID: 5cb769a4cb25f6c811ae2b43035e3227).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.conscrypt/lib64/libjavacrypto.so (Build ID: ba8642c9e08b0b870e8f1a1bda7a9ef8).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.conscrypt/lib64/libssl.so (Build ID: a5a3bc7f8e9d41cf86a435fa84aabcc6).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.runtime/lib64/bionic/libc.so (Build ID: 3707c39fc397eeaa328142d90b50a973).
[495.775] local_symbolizer.cc:481 Could not find /apex/com.android.vndk.v30/lib64/libc++.so (Build ID: 66195581cc855ea56bf2663a42e9115a).
[495.775] local_symbolizer.cc:481 Could not find /system/bin/app_process64 (Build ID: ce3cc65283f87cf0181925988a809dbd).
[495.775] local_symbolizer.cc:481 Could not find /system/framework/x86_64/boot-framework.oat (Build ID: a08fb51e45c8bbd96cf277b15edc84a7eb971431).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libEGL.so (Build ID: 2a2c36a4dd977d11859dfc585e6bf4e1).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libandroid_runtime.so (Build ID: e2556b95e6251c592f076d0623b50aa4).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libandroidfw.so (Build ID: bcb17989bbed4323a42fd500ab4741ab).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libbinder.so (Build ID: f55f2948551e7a9225c5653544eb5571).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libc++.so (Build ID: 5f6fc494386331f011fecfcc8642076c).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libft2.so (Build ID: e404befe9aa2fea418ee0245169a5dbd).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libgui.so (Build ID: 02d9f58fa70710bc6aab51a60a634760).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libhwui.so (Build ID: a588f7ea4e724e9e5d49bce097dee793).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libjpeg.so (Build ID: e746c56e0087900778f27708a9a5ac31).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libminikin.so (Build ID: 31c51bfcba5208f0e17b91bc0195a6a4).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libpng.so (Build ID: 83c7bc99efdcdfd0c7c1fcdb15f8cc89).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libsqlite.so (Build ID: ab3d62ca2c0cb74b4855ae47a6789ca8).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libutils.so (Build ID: 658bede56204d492a28682a718f6f501).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libz.so (Build ID: 727ab7cd735ce4ea52dd3dfa77081220).
[495.775] local_symbolizer.cc:481 Could not find /system/lib64/libziparchive.so (Build ID: 5a97b5974211715f15a4d39ef24ee523).
[495.775] local_symbolizer.cc:481 Could not find /vendor/lib64/egl/libEGL_emulation.so (Build ID: 1c910424fcdb1518e1cd3989660d173c).
[495.775] local_symbolizer.cc:481 Could not find /vendor/lib64/hw/android.hardware.graphics.mapper@3.0-impl-ranchu.so (Build ID: 320d65953c80987dedddf42c7f95067a).
[495.775] local_symbolizer.cc:481 Could not find /vendor/lib64/libGLESv2_enc.so (Build ID: 532afe147eca1b8b8e4dc916da282f18).
[495.775] local_symbolizer.cc:481 Could not find /vendor/lib64/libOpenglCodecCommon.so (Build ID: ca1a7f59d6c91b54f4450902354cce86).
Wrote profiles to /tmp/ad9ea1 (symlink /tmp/heap_profile-latest)
These can be viewed using pprof. Googlers: head to pprof/ and upload them.
fmayer commented 3 years ago

Glad you got this working! Is Android 11 enough for you, or do you need Android 10 on the emulator?

nickaein commented 3 years ago

Yeap, for now we have no problem to use emulator with Android 11 for our profilings. Let me know if there were any need to dig deeper into this issue.

BTW, thank you guys for open sourcing these awesome tools :)

fmayer commented 3 years ago

Thanks for the detailed bug report, glad it's working for you now. Added the x86_64 issues to the Known Issues on Android 10.