lldb python support

[cristianadam]

Qt Creator requires python support from gdb / lldb debugger.

It would be great to have a fully toolchain on Windows 😄

[mstorsjo]

Hmm..after thinking about it, I realized we can build libffi and Python before we build LLVM (for the cross build), and let CMake auto-detect it. So build-llvm.sh can stay unmodified, and we only need a build-python.sh before that.

Yes, that's my idea for it as well. At most you might need some extra lines in build-llvm.sh that check for the presence of the cross compiled python and pass some extra flags to the lldb build to pick it up.

[longnguyen2004]

Adding the Python lib dir to the CMake search path is probably enough. I still have to iron out a couple of issues though, and also try to upstream my Clang support for distutils.

[longnguyen2004]

Wish me luck! https://github.com/pypa/distutils/pull/26/

[RussellHaley]

Nice work! I helped the developer of xmake improve his clang plugins and found a bunch of bugs.

I'm getting really close to having something working. I figured out that python was choking because there was an errant python directory in my appdata/roaming directory. Once that was removed python starts normally.

I also figured out that lldb.exe was exiting because it couldn't find the python dll. Hurray for gdb!

After all this work tonight I understand why the LLDB developers use a shared object and not a static library. It doesn't really seem possible to embed python without the entire library. There seems to be a whole bunch of other "stuff" that is required to use python, not just libpython.

That may be hyperbole, but I'm tired and bitter. I took some stock of the situation and it's not looking good. Between the full python installation and the extra files that LLDB needs, this feature will add 193 MBs to my installer, and it can't be optional. LLDB doesn't work without the python dlls once it's compiled in. I might be able to play some tricks in the MSI and have two lldb executables (depending on the feature level), but my God. This feels very much like tying oneself to a ship anchor and going for a little sail on a stormy day.

I'm used to Lua. A full Lua install is one megabyte. My entire lua + package manager is 7 MB. Even with every LuaRock that I have ever needed I don't think I've had a lua installation over 20 Mb.

Well, you know a developer feels passionate about something if he makes an infographic:

Okay, lunches...

[longnguyen2004]

I might be able to play some tricks in the MSI and have two lldb executables (depending on the feature level)

That's gonna be two LLDB builds that you have to maintain, which is quite a pain.

this feature will add 193 MBs to my installer

A prebuilt Python 3.8 from python.org takes 256MB...I wonder if there's a way we can shrink that down.

While you're working on your WinLua business, I'm gonna write up a draft build_python.sh in my fork, maybe some of you can help me through it. Unfortunately with Python, I don't think we can completely avoid patches. Aside from having to add Clang support to distutils, ~there's another bug that I previously mentioned (/usr/local/lib/pythonX.Y/config-X.Y mistakenly added to the library search path, which screws up the linking step if another libpythonX.Y.a is in there). I tried rebuilding Python in WSL and the issue persists, though there's no libpythonX.Y.a in that folder, so I didn't notice the bug. Probably gonna be another talk with the Python (or MSYS2) devs...~

Turns out it's my fault...if I don't specify a prefix, it defaults to /usr/local...Mystery solved then :)

Alright, time to take a break...

[longnguyen2004]

@mstorsjo For libffi, there's no configure script in the repo, so we have to generate it ourselves, which requires libtool. Can you check if your Docker image contains libtool?

[longnguyen2004]

Here's my current branch which contains a prototype build-python.sh and my distutils patch. If anyone needs to change something then I'll add you as a collaborator.

@mstorsjo For LLVM, I guess adding -DCMAKE_PREFIX_PATH=$PREFIX/lib to cmake is enough. Is there anything in the build system that sets CMAKE_PREFIX_PATH?

[mstorsjo]

@mstorsjo For libffi, there's no configure script in the repo, so we have to generate it ourselves, which requires libtool. Can you check if your Docker image contains libtool?

Yes, they should contain libtool already.

Here's my current branch which contains a prototype build-python.sh and my distutils patch. If anyone needs to change something then I'll add you as a collaborator.

Nice! I'll post a couple random comments on it soon, and try it out for myself later. (Especially compiling for arm might be interesting.)

@mstorsjo For LLVM, I guess adding -DCMAKE_PREFIX_PATH=$PREFIX/lib to cmake is enough. Is there anything in the build system that sets CMAKE_PREFIX_PATH?

I don't think so, grepping for it in my build scripts doesn't find anything at least.

[RussellHaley]

That's gonna be two LLDB builds that you have to maintain, which is quite a pain.

It's not too bad building on the server, but I'm just not keen on this option for the moment.

this feature will add 193 MBs to my installer

A prebuilt Python 3.8 from python.org takes 256MB...I wonder if there's a way we can shrink that down.

While you're working on your WinLua business, I'm gonna write up a draft build_python.sh in my fork, maybe some of you can help me through it.

Awesome. Let me know and can test? That said, you have access to the same server I have. :-)

[longnguyen2004]

Awesome. Let me know and can test? That said, you have access to the same server I have. :-)

Feel free to do your own testing, and inform me of any problems!

For now things seem to be good, except for the import library mismatch issue (TL;DR: distutils appends /usr/local/lib/pythonX.Y/config-X.Y into the library search dirs, and if there happens to be a libpythonX.Y.a, like what I saw on your server, the link step will fail with mysterious "unknown file type" errors). The weird part is that they are using sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")) to check if we're building Python, while just a few lines down they use a much simpler not sysconfig.python_build: 🤔. I'm gonna try discussing it with the Python/MSYS2 devs and see if it's intentional or not.

There's also my clang support patch which I haven't seen any responses from pypa, I'll try submitting it directly to the cpython repo.

Anyway that's it for today, I've spent a few hours grinding away at the script, time to sleep :)

[longnguyen2004]

After having a closer look at the Python source code, I noticed the file Modules/Setup, which allows you to override setup.py and set what modules to build. Upstream Python is also planning to move modules into that eventually. If we can use something like sed to set which modules to build (or ship our own Modules/Setup file), we can bypass setup.py entirely and go back to being patchless 🎉

Still, I submitted my clang support patch to the setuptools repo (see https://github.com/pypa/setuptools/pull/2564), since I figured other users can benefit from my additions.

[longnguyen2004]

Unfortunately that turns out to be a failure. The Modules/Setup file is missing a lot of essentials modules. As such, we can't rely on this, and we have to use the distutils system.

[longnguyen2004]

@RussellHaley Good news! The libffi library works, and _ctypes now builds without a problem. The only failing module is _hashlib, which is related to OpenSSL, so we don't have to worry about it. However on my local WSL build, it doesn't try to build _hashlib at all, so this seems like env pollution, or some FreeBSD Python quirks, or something else. Anyway it's not important for us right now.

The two patches are in my branch, and they'll stay there until either upstream accept it, or distutils got removed.

[RussellHaley]

I'll make a pull request to your llvm-mingw branch. We can debug it from there.

[longnguyen2004]

@mstorsjo Are there any differences between the prefixed windres? AFAIK no other mingw-w64 toolchain has it, and the MSYS2 patches also default to an unprefixed windres.

[mstorsjo]

@mstorsjo Are there any differences between the prefixed windres?

Yes - they produce the resource data packed in an object file, and the object file's architecture must match the architecture of all other object files involved when linking.

AFAIK no other mingw-w64 toolchain has it, and the MSYS2 patches also default to an unprefixed windres.

Well that's wrong from a cross compilation scenario. But as those toolchains don't really cross compile, it's not much of an issue. It looks like both mingw-builds and msys2 only provide the gcc frontends with a triple prefix, but none of the binutils tools. (So in those cases, there's little point to the triple prefix on the compilers either. E.g. if in a msys2/mingw64 setup, if you'd want to run the i686-w64-mingw32 prefixed compilers, your link would fail as it'd refuse to include the x86_64 winres output when linking for i686.) If you look at any distribution of e.g. gcc+binutils for cross compilation, they come with all the binutils tools packaged as <triple>-name, for all of objdump, readelf, strip, objcopy, windres, etc.

When configuring with ./configure --host=<triple>, autotools look for tools in the form of <triple>-toolname, and if not found, fall back on trying plain toolname (which actually is wrong in a number of cases as well, but that's a different story).

I did run into this when I tried to cross compile your branch, but haven't looked at which part of the patches add this and how to fix it cleanly. Ideally it would use the normal autoconf methods for testing for <triple>-toolname, but I haven't had time to dig further into what it would require.

[mstorsjo]

@mstorsjo Are there any differences between the prefixed windres? AFAIK no other mingw-w64 toolchain has it

If you look at any distribution of e.g. gcc+binutils for cross compilation, they come with all the binutils tools packaged as <triple>-name, for all of objdump, readelf, strip, objcopy, windres, etc.

See e.g. https://packages.ubuntu.com/focal/amd64/binutils-mingw-w64-x86-64/filelist

[mstorsjo]

Also fwiw, I got this building for aarch64 with some windres tweaking. But building for armv7 fails in libffi, as I never did upstream the rest of the patches needed for making that work. I guess I should look into that again...

[longnguyen2004]

but haven't looked at which part of the patches add this and how to fix it cleanly

It comes from this patch, which hardcodes the unprefixed windres. I'll bring this up with the MSYS2 folks and maybe get a fix going (or we can just symlink <prefix>-windres to windres when building, and then remove it afterward)

[mstorsjo]

but haven't looked at which part of the patches add this and how to fix it cleanly

It comes from this patch, which hardcodes the unprefixed windres. I'll bring this up with the MSYS2 folks and maybe get a fix going (or we can just symlink <prefix>-windres to windres when building, and then remove it afterward)

Unfortunately, symlinking it doesn't help here, because all the <prefix>-windres ones are just scripts (they're all symlinks to windres-wrapper.sh) that look at $0 to see what they were invoked as - and when you just invoke it as windres, it has no clue what architecture you intended. So for now I worked around it by making windres a minimal shell script that invokes (cd "$(dirname $0)" && pwd)/$HOST-windres "$@".

[longnguyen2004]

Right...I'll report this to MSYS2 then.

[longnguyen2004]

Update: MSYS2 has fixed this in https://github.com/msys2/MINGW-packages/pull/7958 I'm gonna update the script to get the new patches, and also work on the LLVM side.

[longnguyen2004]

SCP file transfer is slow :)

Anyway, I now have the first prototype of the complete toolchain with Python support. I'll upload it here once it finishes transferring.

[longnguyen2004]

Here's the toolchain if anyone wants to experiment with it.

 HP@DESKTOP-194TAE9  C:\..\bin  ./lldb
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> lldb
<module 'lldb' from 'C:/llvm-mingw-python-no-symlink/lib/python3.8/site-packages\\lldb\\__init__.py'>
>>> import sys
>>> sys.version
'3.8.7 (tags/v3.8.7-dirty:6503f05dd5, Feb 14 2021, 18:32:35) \n[Clang 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d45709'
>>>

Now, adding Python to the toolchain will inflate the size by about 200MB. So it's important to consider the benefits vs. the size increase, and how it would affect future releases.

[RussellHaley]

Woot woot! @longnguyen2004 train coming through!

Looking forward to testing with kdevelop...

[RussellHaley]

Here's the toolchain if anyone wants to experiment with it.

 HP@DESKTOP-194TAE9  C:\..\bin  ./lldb
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> lldb
<module 'lldb' from 'C:/llvm-mingw-python-no-symlink/lib/python3.8/site-packages\\lldb\\__init__.py'>
>>> import sys
>>> sys.version
'3.8.7 (tags/v3.8.7-dirty:6503f05dd5, Feb 14 2021, 18:32:35) \n[Clang 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d45709'
>>>

Now, adding Python to the toolchain will inflate the size by about 200MB. So it's important to consider the benefits vs. the size increase, and how it would affect future releases.

I wonder if lldb/python/lldm-mi should be built as a separate package? If the "big-lldb-option" is selected, a second run of cmake that only builds lldb,python,lldb-mi gets selected (maybe a separate shell script)? I don't think the extra 200 MB is worth it for 95% of people that would look to use this toolchain.

[longnguyen2004]

maybe a separate shell script

We don't need to, everything can just go in build-llvm.sh. lldb-mi can be permanently included, since it doesn't take up much space, but we should ask @mstorsjo for that.

I'm gonna work on separating the python part from the rest of the toolchain.

[RussellHaley]

I just installed gcc/mingw from sourceforge. The install puts python in a /opt directory (opt/bin, opt/lib).

[mstorsjo]

maybe a separate shell script

We don't need to, everything can just go in build-llvm.sh. lldb-mi can be permanently included, since it doesn't take up much space, but we should ask @mstorsjo for that.

Yeah lldb-mi is no biggie, that one can go in if it concretely adds some value. (I made scripts for integrating building of it, see the branch lldb-mi in this repo, but when I asked about it at that time, it was unclear if it actually makes some usecase work that didn't work before.)

As for the rest with python and all, I'm clearly intrigued, although it does add a lot of patches and other things... Let's see how neat it becomes and how to integrate it best, and if it's worth making a separate toolchain package including it (for only some architectures and CRT combinations), and if we should keep that in a separate branch, or just in the master branch but built optionally...

And if it'd be possible to only build python+lldb without rebuilding the rest of llvm, as a separate step on top of the regular cross toolchain build, that might be a nice thing to have as well; that'd make it a cheaper thing to build at least, as building all of llvm is pretty massive. But afaik building LLDB requires you to either have a full install of LLVM (including all the individual static libraries of all the internal components, or a shared DLL build with that) so that one can build other apps that call the LLVM APIs, like LLDB, or rebuild all of LLVM. So maybe that doesn't really add so much.

A different question is whether it's possible to make a build of LLDB that optionally uses python, i.e. make some mock python APIs available while building LLDB. But I think that's not supported at the moment - e.g. LLDB from the official LLVM windows releases is built with python, but the python version isn't included, and LLDB doesn't even start unless you have it installed.

[longnguyen2004]

it was unclear if it actually makes some usecase work that didn't work before

I remember having Visual Studio Code work with it before, so it can certainly be useful.

And if it'd be possible to only build python+lldb without rebuilding the rest of llvm

That's why I asked you if we should split lldb into a separate script, since rebuilding llvm is quite a hassle, and we can reuse the llvm build artifact to build lldb.

Let's see how neat it becomes and how to integrate it best

You can check out my prototype toolchain above, it has Python support in LLDB. Currently I'm experimenting with putting the python bit in $PREFIX/python so it can be separated from the rest of the toolchain.

The install puts python in a /opt directory

In this case though, putting it in opt is a bit misleading, because it's not really an optional addon, and if you remove it then the debugger doesn't work. Putting it in $PREFIX/python is good though, and that's what I'm testing.

[RussellHaley]

maybe a separate shell script

We don't need to, everything can just go in build-llvm.sh. lldb-mi can be permanently included, since it doesn't take up much space, but we should ask @mstorsjo for that.

Yeah lldb-mi is no biggie, that one can go in if it concretely adds some value. (I made scripts for integrating building of it, see the branch lldb-mi in this repo, but when I asked about it at that time, it was unclear if it actually makes some usecase work that didn't work before.)

Can lldb-mi be used somewhere without the python integration? I can't remember if I tried it with Eclipse. Why did they peel away lldb-mi into the llvm-tools but keep all the other parts?

As for the rest with python and all, I'm clearly intrigued, although it does add a lot of patches and other things... Let's see how neat it becomes and how to integrate it best, and if it's worth making a separate toolchain package including it (for only some architectures and CRT combinations), and if we should keep that in a separate branch, or just in the master branch but built optionally...

I personally think @longnguyen2004 should separate the python build script and take it over as a 'thing' separate from this repository. I suspect Python with mingw is going to take a lot of long-term maintenance and it could really create a name for him. I'd also really like to see the mingw build usable with llvm-mingw on Windows without MSYS2; but maybe I'll re-visit the cmake ( way, way, way down my list. I just like cool packages associated with this compiler toolchain).

I have only skimmed the article but the LLDB scripting options are very cool: https://lldb.llvm.org/use/python.html

And if it'd be possible to only build python+lldb without rebuilding the rest of llvm, as a separate step on top of the regular cross toolchain build, that might be a nice thing to have as well; that'd make it a cheaper thing to build at least, as building all of llvm is pretty massive. But afaik building LLDB requires you to either have a full install of LLVM (including all the individual static libraries of all the internal components, or a shared DLL build with that) so that one can build other apps that call the LLVM APIs, like LLDB, or rebuild all of LLVM. So maybe that doesn't really add so much.

When changing the python options for the LLDB builds I noticed that cmake/ninja will just rebuild the LLDB files.

A different question is whether it's possible to make a build of LLDB that optionally uses python, i.e. make some mock python APIs available while building LLDB. But I think that's not supported at the moment - e.g. LLDB from the official LLVM windows releases is built with python, but the python version isn't included, and LLDB doesn't even start unless you have it installed.

I think the Lua bindings are a far, far better idea but I do bring some bias to the conversation (Ouch, stop throwing things at me! he he).

Anyway, very cool stuff.

[mstorsjo]

it was unclear if it actually makes some usecase work that didn't work before

I remember having Visual Studio Code work with it before, so it can certainly be useful.

Ah, it was comment discussed here earlier, in https://github.com/mstorsjo/llvm-mingw/issues/73#issuecomment-707531535, that it doesn't add anything of value unless we have python support in place - ok.

And if it'd be possible to only build python+lldb without rebuilding the rest of llvm

That's why I asked you if we should split lldb into a separate script, since rebuilding llvm is quite a hassle, and we can reuse the llvm build artifact to build lldb.

Sure - for that use case, being able to build LLDB separately would be useful.

Unfortunately, it still requires you to either have a full install of LLVM to be able to build LLDB (i.e. not with LLVM_INSTALL_TOOLCHAIN_ONLY), or keep the dirty build tree from building LLVM to build on top of it, otherwise building LLDB requires rebuilding all of LLVM again. One can't just take the regular toolchain release and build LLDB with it, without essentially rebuilding all of LLVM.

Let's see how neat it becomes and how to integrate it best

You can check out my prototype toolchain above, it has Python support in LLDB. Currently I'm experimenting with putting the python bit in $PREFIX/python so it can be separated from the rest of the toolchain.

That would be nice if it's possible!

And if it'd be possible to only build python+lldb without rebuilding the rest of llvm, as a separate step on top of the regular cross toolchain build, that might be a nice thing to have as well; that'd make it a cheaper thing to build at least, as building all of llvm is pretty massive. But afaik building LLDB requires you to either have a full install of LLVM (including all the individual static libraries of all the internal components, or a shared DLL build with that) so that one can build other apps that call the LLVM APIs, like LLDB, or rebuild all of LLVM. So maybe that doesn't really add so much.

When changing the python options for the LLDB builds I noticed that cmake/ninja will just rebuild the LLDB files.

Sure - if you have a previously fully built tree in place (which is a number of gigabytes); if you don't, building LLDB requires building all of LLVM again.

[longnguyen2004]

Can lldb-mi be used somewhere without the python integration?

I remember testing it with Visual Studio Code and it works fine.

Why did they peel away lldb-mi into the llvm-tools but keep all the other parts?

Probably because there is a much better way to control the debugger: The Debug Adapter Protocol (Thanks Microsoft!). LLDB added an implementation for it in this commit (don't let the name fool you, it can be used with any IDE that implements the protocol). Many IDE has also implemented it (see the list of supported IDE).

I'd also really like to see the mingw build usable with llvm-mingw on Windows without MSYS2

If you're talking about MSYS2 the POSIX environment, then it's not dependent on any MSYS2 component. All I've done is use the MSYS2 patchset which they maintain. It's way easier to just reuse code that has been tested instead of coming up with our own.

[mstorsjo]

I'd also really like to see the mingw build usable with llvm-mingw on Windows without MSYS2

If you're talking about MSYS2 the POSIX environment, then it's not dependent on any MSYS2 component. All I've done is use the MSYS2 patchset which they maintain. It's way easier to just reuse code that has been tested instead of coming up with our own.

I think his complaint is that it uses autotools build scripts, which requires you to either cross compile or be building it within MSYS2 (or similar), even though the build itself is a fully mingw build with no reliance on posix emulation layers like msys/cygwin.

[mstorsjo]

Can lldb-mi be used somewhere without the python integration?

I remember testing it with Visual Studio Code and it works fine.

Hmm, ok, so I asked about it in https://github.com/mstorsjo/llvm-mingw/issues/55#issuecomment-706368646 and didn't get followup there, and https://github.com/mstorsjo/llvm-mingw/issues/73#issuecomment-707508475 indicated that lldb-mi wasn't usful as such, so that's why I left it out from the last release.

Why did they peel away lldb-mi into the llvm-tools but keep all the other parts?

Probably because there is a much better way to control the debugger: The Debug Adapter Protocol (Thanks Microsoft!). LLDB added an implementation for it in this commit (don't let the name fool you, it can be used with any IDE that implements the protocol). Many IDE has also implemented it (see the list of supported IDE).

Ok, so would including lldb-vscode be useful (as it's trivial to do)? We discussed this back in https://github.com/mstorsjo/llvm-mingw/issues/83#issuecomment-582924233 and it seemed from that comment that it wasn't necessary and didn't add any extra value - do you still agree or should we enable including that as part of the default build?

[longnguyen2004]

Update: I have separated the python bit into $PREFIX/python and the debugger still works

(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> lldb
<module 'lldb' from 'C:/llvm-mingw-python-sep-no-symlink/python/lib/python3.8/site-packages\\lldb\\__init__.py'>
>>> sys.version
'3.8.7 (tags/v3.8.7-dirty:6503f05dd5, Feb 14 2021, 23:18:44) \n[Clang 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d45709'
>>> sys.path
['C:/llvm-mingw-python-sep-no-symlink/bin', 'C:/llvm-mingw-python-sep-no-symlink/python/lib/python3.8/site-packages', 'C:\\llvm-mingw-python-sep-no-symlink\\python\\lib\\python38.zip', 'C:\\llvm-mingw-python-sep-no-symlink\\python\\lib\\python3.8', 'C:\\llvm-mingw-python-sep-no-symlink\\python\\lib\\python3.8\\lib-dynload', 'C:\\Users\\*****\\AppData\\Roaming\\Python\\Python38\\site-packages', 'C:\\llvm-mingw-python-sep-no-symlink\\python', '.']
>>>

So, I have the crazy idea of removing the python folder while leaving the dlls with lldb. The debugger still runs without a problem, but if I try to enter script mode, it crashes

(lldb) script
Python path configuration:
  PYTHONHOME = 'C:\llvm-mingw-python-sep-no-symlink\bin\../python'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = 'C:\\llvm-mingw-python-sep-no-symlink\\bin\\lldb.exe'
  sys.base_prefix = 'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python'
  sys.base_exec_prefix = 'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python'
  sys.executable = 'C:\\llvm-mingw-python-sep-no-symlink\\bin\\lldb.exe'
  sys.prefix = 'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python'
  sys.exec_prefix = 'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python'
  sys.path = [
    'C:\\llvm-mingw-python-sep-no-symlink\\bin\\..\\python\\lib\\python38.zip',
    'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python\\lib\\python3.8',
    'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python\\lib\\python3.8',
    'C:\\llvm-mingw-python-sep-no-symlink\\bin\\../python\\lib\\python3.8\\lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000026ac (most recent call first):
<no Python frame>

So, aside from the somewhat hacky method, we can use this to remove Python support without rebuilding lldb.

Another method is to zip up the Python stdlib and let Python unpack it at runtime. This will drop the size increase of the toolchain to ~100MB. But we'll have to build Python with zlib, so it can unpack the zip file.

which requires you to either cross compile or be building it within MSYS2 (or similar)

Our build scripts are also shell scripts, so we have to use MSYS2 on Windows in the first place.

do you still agree or should we enable including that as part of the default build?

I guess I need to do some more testing with both of them to see if it can be valuable.

[mstorsjo]

Update: I have separated the python bit into $PREFIX/python and the debugger still works

(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> lldb
<module 'lldb' from 'C:/llvm-mingw-python-sep-no-symlink/python/lib/python3.8/site-packages\\lldb\\__init__.py'>
>>> sys.version
'3.8.7 (tags/v3.8.7-dirty:6503f05dd5, Feb 14 2021, 23:18:44) \n[Clang 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d45709'
>>> sys.path
['C:/llvm-mingw-python-sep-no-symlink/bin', 'C:/llvm-mingw-python-sep-no-symlink/python/lib/python3.8/site-packages', 'C:\\llvm-mingw-python-sep-no-symlink\\python\\lib\\python38.zip', 'C:\\llvm-mingw-python-sep-no-symlink\\python\\lib\\python3.8', 'C:\\llvm-mingw-python-sep-no-symlink\\python\\lib\\python3.8\\lib-dynload', 'C:\\Users\\*****\\AppData\\Roaming\\Python\\Python38\\site-packages', 'C:\\llvm-mingw-python-sep-no-symlink\\python', '.']
>>>

So, I have the crazy idea of removing the python folder while leaving the dlls with lldb. The debugger still runs without a problem, but if I try to enter script mode, it crashes

Nice, that also sounds like a very viable option.

[longnguyen2004]

More progress: So I tested lldb and lldb-mi/lldb-vscode with a couple of IDEs and it works like a charm

Eclipse (lldb-mi - Call Stack, Breakpoints, Variables working)

Visual Studio Code (lldb-mi - Call Stack, Breakpoints, Variables working)

I will update this comment as I go along.

[RussellHaley]

I tried KDevelop using @longnguyen2004s python-lldm-mi build but it didn't work: The LLDB console gives the following errors:

C:/Users/russh/llvm-mingw-w64-x86_64/bin/lldb-mi.exe error: module importing failed: invalid pathname error: module importing failed: invalid pathname error: module importing failed: invalid pathname error: module importing failed: invalid pathname error: 'settings set' takes more arguments error: 'settings set' takes more arguments error: 'settings set' takes more arguments

(Edit: Could it be that KDevelop is expecting python 2?)

[RussellHaley]

@mstorsjo do you know if LLDB still confined to 32bit applications on Windows?

[mstorsjo]

@mstorsjo do you know if LLDB still confined to 32bit applications on Windows?

When has that been the case? A 32 bit build of LLDB only can debug 32 bit processes though. A 64 bit LLDB might be able to debug a 32 bit process but that hasn't necessarily worked properly in my experience, but as long as LLDB and the process is of the same architecture, it should be fine for both 32 and 64 bit.

[longnguyen2004]

I tried KDevelop using @longnguyen2004s python-lldm-mi build but it didn't work

I never got KDevelop to work, maybe it was expecting Python 2, not 3, but I don't have time to investigate it, given that lldb-mi works great in both Eclipse and Visual Studio Code.

Also I still haven't got around to uploading my python separated toolchain, I'll try to find some time to do it today. Update: This is the toolchain (removed a few non-essential modules, such as test, tkinter, sqlite3, curses, which takes up quite a lot of space)

[RussellHaley]

@mstorsjo do you know if LLDB still confined to 32bit applications on Windows?

When has that been the case? A 32 bit build of LLDB only can debug 32 bit processes though. A 64 bit LLDB might be able to debug a 32 bit process but that hasn't necessarily worked properly in my experience, but as long as LLDB and the process is of the same architecture, it should be fine for both 32 and 64 bit.

It says that on the lldb home page? Or am I misunderstanding? https://lldb.llvm.org/

Windows user-space debugging for i386 (*)
(*) Support for Windows is under active development. Basic functionality is expected to work, with functionality improving rapidly.

@longnguyen2004 are you using a plugin? Could you share your launch configuration?

[RussellHaley]

Can lldb-mi be used somewhere without the python integration?

I remember testing it with Visual Studio Code and it works fine.

Hmm, ok, so I asked about it in #55 (comment) and didn't get followup there, and #73 (comment) indicated that lldb-mi wasn't usful as such, so that's why I left it out from the last release.

Why did they peel away lldb-mi into the llvm-tools but keep all the other parts?

Probably because there is a much better way to control the debugger: The Debug Adapter Protocol (Thanks Microsoft!). LLDB added an implementation for it in this commit (don't let the name fool you, it can be used with any IDE that implements the protocol). Many IDE has also implemented it (see the list of supported IDE).

Ok, so would including lldb-vscode be useful (as it's trivial to do)? We discussed this back in #83 (comment) and it seemed from that comment that it wasn't necessary and didn't add any extra value - do you still agree or should we enable including that as part of the default build?

This conversation has become quite long and hard to follow. I think adding python 2 support would be largely trivial at this point if it's necessary for qt Creator or KDevelop. Perhaps since @longnguyen2004 has largely solved this problem, we close the issue and open a new one and inventory what we know works and what doesn't work (debugger interfaces vis-a-vis IDEs)?

[mstorsjo]

@mstorsjo do you know if LLDB still confined to 32bit applications on Windows?

When has that been the case? A 32 bit build of LLDB only can debug 32 bit processes though. A 64 bit LLDB might be able to debug a 32 bit process but that hasn't necessarily worked properly in my experience, but as long as LLDB and the process is of the same architecture, it should be fine for both 32 and 64 bit.

It says that on the lldb home page? Or am I misunderstanding? https://lldb.llvm.org/

Windows user-space debugging for i386 (*)
(*) Support for Windows is under active development. Basic functionality is expected to work, with functionality improving rapidly.

Oh, ok - I'd say that is pretty outdated. I'll see about updating it.

[RussellHaley]

In case anyone is interested in the lldb-vscode extension (quite off topic), there is a readme on how to configure it in the code tree:

https://github.com/llvm/llvm-project/blob/main/lldb/tools/lldb-vscode/README.md

[longnguyen2004]

Alright, someone creates a new issue and I'll write everything I know about it.

[mstorsjo]

@mstorsjo do you know if LLDB still confined to 32bit applications on Windows?

When has that been the case? A 32 bit build of LLDB only can debug 32 bit processes though. A 64 bit LLDB might be able to debug a 32 bit process but that hasn't necessarily worked properly in my experience, but as long as LLDB and the process is of the same architecture, it should be fine for both 32 and 64 bit.

It says that on the lldb home page? Or am I misunderstanding? https://lldb.llvm.org/

Windows user-space debugging for i386 (*)
(*) Support for Windows is under active development. Basic functionality is expected to work, with functionality improving rapidly.

https://github.com/llvm/llvm-project/commit/ae14f3fdbfa85c92d64f5f38f883a6db4266ca80

(Let's see how long it takes before the website updates.)

[mstorsjo]

FYI I picked up the build-python branch that @longnguyen2004 had done, squashed it and did a few minor improvements:

• Tried to shrink the footprint of the python installation
• Integrated building into Dockerfile.cross
• Building a native version of python first (as cross compiling python requires you to have the same minor version of python available natively too - this fixes building e.g. python 3.8 on ubuntu 18.04 which ships with python 3.6)
• Updated to a newer version of libffi that supports mingw-armv7 (so the same can be built for all 4 architectures)
• Squashed the patches to a neat series that is easier to rebase
• Rebased on top of the latest master version, including LLVM 12.0.0
• Included lldb-mi

See https://github.com/mstorsjo/llvm-mingw/commits/python for this series.

I built https://martin.st/temp/llvm-mingw-lldbtest-x86_64.zip and https://martin.st/temp/llvm-mingw-lldbtest-aarch64.zip with it.

[longnguyen2004]

Thank you for picking up my Python work and improving on it! Most of the work has been done so it wouldn't be too hard to update Python in the future. If you have any questions, I'll be glad to answer them. In the future, when Python fully switches to Makefiles to build, we won't have to build a native Python anymore, which is a huge time saver.