search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
28.62k stars 11.83k forks source link

LLDB doesn't upload executable in remote debugging #56927

Open sadilekivan opened 2 years ago

sadilekivan commented 2 years ago

I'm trying to do a workaround with platform put-file, but LLDB should upload the specified target automatically. Can anyone try to reproduce this? Running lldb version 14.0.0.

(lldb) platform select remote-linux
  Platform: remote-linux
 Connected: no
(lldb) platform connect connect://pidevel.local:15555
  Platform: remote-linux
    Triple: aarch64-unknown-linux-gnu
OS Version: 5.15.32 (5.15.32-v8+)
  Hostname: pidevel
 Connected: yes
WorkingDir: /home/tcele
    Kernel: #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022
(lldb) file target/aarch64-unknown-linux-gnu/debug/examples/simple
Current executable set to '/home/ivan/workspace/TCELE/pitron-core/target/aarch64-unknown-linux-gnu/debug/examples/simple' (aarch64).
(lldb) target list
Current targets:
* target #0: /home/ivan/workspace/TCELE/pitron-core/target/aarch64-unknown-linux-gnu/debug/examples/simple ( arch=aarch64-*-linux, platform=remote-linux )
(lldb) run
error: No such file or directory

If I upload the file (to /home/tcele/simple) myself, all works fine and I can debug.

llvmbot commented 2 years ago

@llvm/issue-subscribers-lldb

DavidSpickett commented 2 years ago

I'm trying to do a workaround with platform put-file

Did that work for you? I have had issues with permissions on the remote before, though I seem to remember getting an error specific to that.

There may be a logging channel that would give some useful info. log enable <tab> to autocomplete the list. Anything related to platform or process might help.

I will try this myself tomorrow.

sadilekivan commented 2 years ago

Did that work for you?

(lldb) platform put-file Cargo.lock Cargo.lock
error: No such file or directory

So far it has not. I have access to the working directory, so hopefully not a silly permission issue. I will check out the log you suggested

sadilekivan commented 2 years ago

Seems like I might be getting somewhere:

(lldb) platform put-file Cargo.lock Cargo.lock
lldb             Processing command: platform put-file Cargo.lock Cargo.lock
lldb             HandleCommand, cmd_obj : 'platform put-file'
lldb             HandleCommand, (revised) command_string: 'platform put-file Cargo.lock Cargo.lock'
lldb             HandleCommand, wants_raw_input:'False'
lldb             HandleCommand, command line after removing command name(s): 'Cargo.lock Cargo.lock'
lldb             [PutFile] Using block by block transfer....

lldb             dest_file = 18446744073709551615

lldb             HandleCommand, command did not succeed
error: No such file or directory
DavidSpickett commented 2 years ago

I didn't see the issue if I started a platform locally then connected to that,but in my qemu-system setup I was able to reproduce it.

Edit: Turns out didn't manage to reproduce it, just a genuinely missing file I happened to choose.

DavidSpickett commented 2 years ago

I did find something useful though. We know from that logging you at least got to https://github.com/llvm/llvm-project/blob/1bd31a689844a43c9cde41a067119bef11159539/lldb/source/Target/Platform.cpp#L1205.

So either we failed to get permissions (GetPermissions takes an error but we don't check it?) or failed to open for some other reason. I'm thinking the latter because source should be the local file while dest_file is on the remote.

UINT64_MAX certainly means we failed to open the file. len(bin(18446744073709551615)[2:]) is 64 so it's UIN64_MAX.

Where we go from there I'm not sure. Are you able to rebuild lldb? You could add more logging (/printfs) to this to figure out what it's being asked to open. Also try building from main to see if this has been fixed since.

DavidSpickett commented 2 years ago

Here is what I get from my working setup which is using lldb from main and connecting to a qemu-system remote.

(lldb) log enable lldb platform
(lldb) target create ./lldb-test-build.noindex/linux/aarch64/mte_memory_region/TestAArch64LinuxMTEMemoryRegion.test_mte_regions/a.out
lldb             PlatformRemoteGDBServer::GetModuleSpec - failed to get module info for /work/open_source/lldb-cross-compile/build-host/lldb-test-build.noindex/linux/aarch64/mte_memory_region/TestAArch64LinuxMTEMemoryRegion.test_mte_regions/a.out:aarch64--linux
lldb             PlatformRemoteGDBServer::GetModuleSpec - failed to get module info for libc.so.6:aarch64--linux
Current executable set to '/work/open_source/lldb-cross-compile/build-host/lldb-test-build.noindex/linux/aarch64/mte_memory_region/TestAArch64LinuxMTEMemoryRegion.test_mte_regions/a.out' (aarch64).
(lldb) run
lldb             target 0x00000000014F1720
lldb             PlatformRemoteGDBServer::GetRemoteWorkingDirectory() -> '/mnt/virt_root'
lldb             Platform::Install (src='/work/open_source/lldb-cross-compile/build-host/lldb-test-build.noindex/linux/aarch64/mte_memory_region/TestAArch64LinuxMTEMemoryRegion.test_mte_regions/a.out', dst='/mnt/virt_root/a.out')
lldb             Platform::Install (src='/work/open_source/lldb-cross-compile/build-host/lldb-test-build.noindex/linux/aarch64/mte_memory_region/TestAArch64LinuxMTEMemoryRegion.test_mte_regions/a.out', dst='/mnt/virt_root/a.out') fixed_dst='/mnt/virt_root/a.out'
lldb             [PutFile] Using block by block transfer....

lldb             dest_file = 5

lldb             PlatformRemoteGDBServer::SetFilePermissions(path='/mnt/virt_root/a.out', file_permissions=700) error = 0 ((null))
lldb             ProcessGDBRemote::GetModuleSpec - got module info for (/usr/lib/aarch64-linux-gnu/ld-2.31.so:aarch64-unknown-linux-gnu) : file = '/usr/lib/aarch64-linux-gnu/ld-2.31.so', arch = aarch64-unknown-unknown, uuid = F80E9BF9-91A5-71A7-74F8-B9F28971B0A5-8CC2B240, object size = 146320
Process 34768 launched: '/work/open_source/lldb-cross-compile/build-host/lldb-test-build.noindex/linux/aarch64/mte_memory_region/TestAArch64LinuxMTEMemoryRegion.test_mte_regions/a.out' (aarch64)

Probably not going to show you anything new but at least you could compare which path should be host and remote.

Another thing I thought of is you could try gdb to gdbserver. Since it will be doing it in a very similar way so if that fails, we could be looking at an issue on the remote itself.

rmoats commented 2 years ago

I am seeing this with macOS lldb-1316.0.9.46 connected to a remote linux lldb-14.0.0 — it appears to be the case that the target file will only be written if it already exists on the remote. Just touching the remote file allows lldb to write to it.

[update] it also turns my 97M executable into a 391M executable on the remote; this executable will start, but fails immediately with a SEGV

[update 2] it all works as expected when using lldb-14.0.6 on the macOS host side...

DavidSpickett commented 2 years ago

Do you/does anyone know how lldb-1316.0.9.46 maps to upstream llvm? I assume it is the system lldb on a Mac, based on lldb 13?

I can try that on my setup, previously I only tried matched versions.

rmoats commented 2 years ago

Do you/does anyone know how lldb-1316.0.9.46 maps to upstream llvm? I assume it is the system lldb on a Mac, based on lldb 13?

It's the one that ships with Xcode 13.4.1, but figuring out how it maps to upstream seems to be non-trivial...

jasonmolenda commented 2 years ago

Do you/does anyone know how lldb-1316.0.9.46 maps to upstream llvm? I assume it is the system lldb on a Mac, based on lldb 13?

I can try that on my setup, previously I only tried matched versions.

This is the lldb in Xcode 13.4.1, part of the swift-5.6 release. It's based on this branch from the llvm.org repository tagged 2021-10-26: https://github.com/apple/llvm-project/tree/stable/20211026 but this repository has additional support for Swift added in.

The actual lldb binary in that Xcode has some minor patches on top of the github repository, but they tend to be very minor/unusual/internal things that external-to-apple people would not hit. Anything that can be upstreamed to llvm.org or the github swift branches, is, or will be as soon as we get it done. ;)

FWIW the lldb testsuite can run entirely using the platform packets to a remote system via the SB API, but the last time I looked, it wasn't nearly as straightforward to upload a binary, set the permissions correctly, and run it on a remote system using the command line interface in lldb. There was also a handful of lldb bugs in its implementation of the platform packets -- lldb would only interop correctly with lldb-server where the implementation was self consistent. mgorny fixed a bunch of these last year, I think they should have been in that branch e.g. https://reviews.llvm.org/D107475 https://reviews.llvm.org/D106985 https://reviews.llvm.org/D106984 . Last I looked, turning on the gdb-remote packet log in lldb log enable gdb-remote packets didn't show you platform communciation. I wrote a little platform server for darwin systems and added logging on that side for an easy way to see what was being sent/received between the two. Maybe it's easier to turn on the logs on your remote platform stub.

jasonmolenda commented 2 years ago

(nb this branch in apple/llvm-project won't build by itself; it has to be built with swift and a bunch of other repositories, I could outline all of that -- but if this was me, I'd start with packet logs, honestly. platform packets are very easy to read.)