Python3 support in GDB

[Novakov]

GDB included in toolchain package is compiled with Python 2 support. It would be nice to enable support for Python 3 as support for 2.7 is ending next year.

I don't know if I should raise this issue here or in arm-none-eabi-gcc-build repository. If in the -build one, let me know I will create issue there.

Moved from https://github.com/gnu-mcu-eclipse/arm-none-eabi-gcc/issues/5

[ilg-ul]

I'll try to add support for Python 3 in the next release, as a new binary, like *-gdb-py3.

[Novakov]

Great, thanks!

[ilg-ul]

I did some changes in the build scripts to improve python support.

unfortunatelly my experience with gdb-py is close to none, and I'm not sure the chages are ok.

for example, the new scripts will copy the python shared libraries in the same folder where gdb-py is located, such that gdb will always start, and not complain that python is not installed.

however I have no idea if python support is really functional in this configuration. I also have no idea what happens if the installed python has a different version (assuming a different 2.x) than the version included in gdb (2.7.13).

I would appreciate some help to test the new gdb support, or at least some feedback.

in the next few days I'll try to make a pre-release with the new configuration, and possibly add gdb-py3 to the set of binaries.

[Novakov]

Sure thing. Once binaries are released I will check how they works with py2/3 support (at least on Windows, maybe on Linux)

[ilg-ul]

I tryed to create a gdb-py3, but I got:

checking whether to use python... /Users/ilg/opt/homebrew/xbb/bin/python3
checking for python3.7... no
configure: error: no usable python found at /Users/ilg/opt/homebrew/xbb/bin/python3

my python3 config is:

ilg-mbp:~ ilg$ which python3
/Users/ilg/opt/homebrew/xbb/bin/python3
ilg-mbp:~ ilg$ python3 --version
Python 3.7.2

the script uses exactly the same configuration, I only replaced python2 with python3.

I'm not sure, but the gdb configure might not be ready for python3.

do you know of any distribution providing a python3 enabled gdb?

[Novakov]

Both native gdb and arm-none-eabi-gdb on Arch Linux works with Python3.7. Crosstool-ng also seems confirm that gdb supports Python3 https://github.com/crosstool-ng/crosstool-ng/blob/master/config/debug/gdb.in.cross#L53

[ilg-ul]

ok, then it should work for us too; probaby I have to do something different.

[Novakov]

Arch Linux has easily available scripts they are using to built packages. Here is one for GDB: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/arm-none-eabi-gdb

[ilg-ul]

python3 also requires libintl, from the gettext package. I added it and the configure+make were successful.

later today I'll try to create some pre-release binaries and allow you to test them.

[ilg-ul]

unfortunately building the windows gdb-py3 is more complicated than anticipated.

for the linux/mac versions, configure calls the python binary with various options to discover configuration strings, like include paths, lib paths and cflags.

for the windows version using python2, it is necessary to download the .msi file, unpack it with 7za, and use a script that pretends to be python, to return the paths to the folder containing the unpacked .msi and other settings.

in principle the same trick can be done for python3, but in the standard python distribution there is no .msi any more, now they provide .exe files, and I don't know how to extract the content from them. I tried the same 7za, but got some usable files.

I also checked the .zip files advertised for embedded use, but they do not include the headers.

the only place I found the headers is the source archive, but the headers cannot be used as is, they need to be processed by configure, for example to transform pyconfig.h.in -> pyconfig.h.

the conclusion is that, as usual, building for windows is hell.

if you know a method to extract the content from the https://www.python.org/ftp/python/3.7.2/python-3.7.2-amd64.exe file, please let me know, otherwise I'm afraid for now I cannot add gdb-py3 for windows.

[Novakov]

Ok, I found something that may work: on https://www.python.org/ftp/python/3.7.2/amd64/ there are a lot of MSI files for different parts of python install. dev.msi contains includes and libraries.

[ilg-ul]

thank you, I'll take a look. did you manage to extract the content with 7za? was pyconfig.h there?

[Novakov]

Here's list of files extracted from that archive: https://gist.github.com/Novakov/90e6c33a37187c90df81a426c9ac64c7 Sadly directories are transformed into subdirectory/filename -> subdirectory_filename when using 7z

[ilg-ul]

if 7za fails, can you suggst other tool that I can compile on linux to extract the content of msi files?

[Novakov]

7z (not 7za) is far from failing ;) I put together some script to unpack archive and normalize files - see linked gist (files unpack.sh and unpacked.txt)

[ilg-ul]

do you have a link where I can find the sources for this miraculous 7z? (personally I'm not comfortable with these non-unix solutions).

[Novakov]

It seems to be packaged with 7z in p7zip-full package.

[ilg-ul]

not very helpful. my production build machines are CentOS 6 and macOS 10.10, but on top of them I build all modern tools from sources.

even finding the 7za sources was an adventure in itself. I'll try that program on the new msi files, but I'm not very optimistic on the outcome.

[Novakov]

Ok, now I see the issue (I'm not familiar with details of your build process/infrastructure). I took the p7zip sources (https://sourceforge.net/projects/p7zip/files/p7zip/16.02/), built 7z target (make 7z, by default only 7za is built) and executed bin/7z l dev.msi which listed files in the archive confirming that it works.

[ilg-ul]

ah, ok, I know that awful project. :-(

I'll try to build the full 7z, but I rememebr last time I had some problems and gave up with only 7za.

[ilg-ul]

It took me quite a while to put the tools together and now I have a new build environment which also includes 7z.

however I'm not able to extract the files from https://www.python.org/ftp/python/3.7.2/python-3.7.2-amd64.exe, all I get is

$ ls -l tmp/x
total 872
-rw-r--r-- 1 ilg ilg  86039 Dec 23 22:14 0
-rw-r--r-- 1 ilg ilg 617472 Dec 23 21:25 u0
-rw-r--r-- 1 ilg ilg  12028 Dec 23 21:16 u1
-rw-r--r-- 1 ilg ilg  10469 Dec 23 21:16 u2
-rw-r--r-- 1 ilg ilg  57891 Dec 23 21:16 u3
-rw-r--r-- 1 ilg ilg  97806 Dec 23 22:14 u4

I tried the Ubuntu 7z and 7za and none worked. the 7z that I compiled behaves the same.

the problem is that we need both the headers and the precompiled Windows dll. I can find the headers in https://www.python.org/ftp/python/3.7.2/python-3.7.2.post1-embed-amd64.zip, but the dll is not there.

as a workaround for tests I'll try to install the exe on windows and steal the dll from there, but we need a proper solution.

[ilg-ul]

oops! my bad :-(

python-3.7.2.post1-embed-amd64.zip includes the DLLs, and I also found the header files in the main source archive.

now I need to convince configure to accept them in separate locations...

[ilg-ul]

I finally found a workaround for both the DLLs and headers, and solved most of the small configurations problems, but in the end I could not make it build for windows, GDB does not support the new Python3 API when building with mingw-w64:

https://sourceware.org/bugzilla/show_bug.cgi?id=24469

the macOS and Linux builds seem ok.

I'm not very comfortable to make a release with partial support, I'll probably mark it experimental.

[ilg-ul]

Experimental support added in 8.2.1-1.6.

We'll leave this ticket open until fully functional.