search

Nuitka / Nuitka

Nuitka is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4-3.12. You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.
http://nuitka.net
Apache License 2.0
12.06k stars 651 forks source link

Add support for MSYS2 based Python #217

Closed arkdlite closed 5 years ago

arkdlite commented 5 years ago

Hello! I am going to compile a helloworld script in MSYS2 on Windows x64. I have installed package mingw-w64-x86_64-python3 and Nuitka from git (I tested 0.6.1 and 0.6.2rc1). When I run python3 -m nuitka test.py --mingw I get: Error, cannot find 'python37m.lib' file. I found this file in MSVC compiled Python's directory but I can't use it there. I cannot use MSVC compiled Python because I am going to use a package for Python written in C99 and using some POSIX libraries.

Thanks for help!

kayhayen commented 5 years ago

I will try and reproduce it over the weekend and see if it's something easy to fix. I think I already have MSYS2 installed because it is the only sane way to have ccache.exe. There is also the issue of me wanting static linking on Windows too, which might be more possible there.

Nuitka has succeeded in creating extension modules that load into MSVC compiled CPython and for Anaconda, it has used both MSVC and MinGW64 compiled extension modules, for both the MinGW64 compiled (default) and a MSVC compiled variant. So I am not sure I believe strictly in the need to stick to same compilers there.

kayhayen commented 5 years ago

@arkdlite I just succeeded in compiling with msys2, and more or less out of the box. It complained about missing crypt.h in the Python header file, so I did this:

pacman -S libcrypt-devel

I already had those, not sure if it's automatic, or if I had sone those myself:

pacman -Qe | grep gcc
gcc 7.4.0-1
gcc-libs 7.4.0-1

And then this worked:

python bin/nuitka-run tests/basics/Asserts.py

But this one does not right away, although I think this may well have to do with exported symbols from the library, and could be fixable in Nuitka:

python2 bin/nuitka-run tests/basics/Asserts.py

Asserts.build/CompiledCellType.o:CompiledCellType.c:(.rdata$.refptr._PyGC_generation0[.refptr._PyGC_generation0]+0x0): Warnung: undefinierter Verweis auf »_PyGC_generation0«

Also nice, gcc is picked up from PATH automatically, so this is also an easy way to bootstrap things. Now let me check if that Python2 issue is easy to get by.

Yours, Kay

kayhayen commented 5 years ago

This is the kind of diff that makes it work (ignore the Python3.7 part for now)

-#if defined(_WIN32) || PYTHON_VERSION >= 370
+#if defined(_WIN32) || defined(__MSYS__) || PYTHON_VERSION >= 370

It seems that compiler does not define _WIN32 which Nuitka is however using all over the map. For that reason, I believe it will likely not work in standalone mode, or maybe it will, because the #else is typically posix code, and that's good.

I will eventually run the test suite in this setup.

kayhayen commented 5 years ago

I think I want to have #218 before this definitely. It should make standalone much faster, and potentially solves the issue of having to parse dependency walker output by not using it, but instead using pefile module instead.

arkdlite commented 5 years ago

@kayhayen Thanks for our help! I launched MSYS2 console (before I used MINGW-x64 console) and installed these packages:

python3
python3-pip
gcc
libcrypt-devel

Then I was installed stable Nuitka via pip python3 -m pip install nuitka When I run python3 -m nuitka --standalone test.py I get:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/nuitka/__main__.py", line 188, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/nuitka/__main__.py", line 182, in main
    MainControl.main()
  File "/usr/lib/python3.7/site-packages/nuitka/MainControl.py", line 852, in main
    standalone_entry_points = standalone_entry_points
  File "/usr/lib/python3.7/site-packages/nuitka/freezer/Standalone.py", line 1154, in copyUsedDLLs
    used_dlls = detectUsedDLLs(source_dir, standalone_entry_points)
  File "/usr/lib/python3.7/site-packages/nuitka/freezer/Standalone.py", line 1047, in detectUsedDLLs
    for binary_filename, used_dlls in waitWorkers(workers):
  File "/usr/lib/python3.7/site-packages/nuitka/utils/ThreadedExecutor.py", line 33, in waitWorkers
    yield future.result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.7/site-packages/nuitka/freezer/Standalone.py", line 1029, in addDLLInfo
    package_name       = package_name
  File "/usr/lib/python3.7/site-packages/nuitka/freezer/Standalone.py", line 1019, in detectBinaryDLLs
    assert False, Utils.getOS()
AssertionError: MSYS_NT-10.0

But I found test.dist folder and I copied into this:

msys-2.0.dll
msys-iconv-2.dll
msys-intl-8.dll
msys-python-3.7m.dll

Finally, helloworld works, but I see a lot of .so files in the project directory and when I deleted them .exe file worked normally. Now I will try to import some libs. Thanks!

kayhayen commented 5 years ago

I got changes not yet pushed to the public that deal with the OS things that are seen in standalone mode.

As for dependency walker or pefile, I was considering if we couldn't use ldd which probably also exists, just like we do on Linux, and then it's unrelated even. I will try it out once I find the time, and polish the other changes to be more specific about "Windows" vs. "Win32" needs.

I am almost sure it's likely we add MSYS2 support in the next release. I only found that the installation of MSYS2 a bid tedious and confusing for normal Windows folks. But my hope is for static linking to be possible there, and then it could be the fastest CPython/Nuitka on Windows.

kayhayen commented 5 years ago

So this looks really good:

$ ldd Asserts.bin
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff822430000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff820130000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ff81ea20000)
        msys-2.0.dll => /usr/bin/msys-2.0.dll (0x180040000)
        msys-python2.7.dll => /usr/bin/msys-python2.7.dll (0x46b270000)

So standalone mode the POSIX way should be just working, very nice. I don't have the time right now, but surely this will come.

kayhayen commented 5 years ago

So on factory there is now changes that make accelerated binaries work. Didn't try extension modules, and know that standalone needs changes, but I don't want to conflict with the mentioned PR, so I am waiting for its merge, there is no hurry for this.

kayhayen commented 5 years ago

So things but standalone should work on develop now, checking out how standalone based on ldd.exe could be done.

kayhayen commented 5 years ago

I just managed to succeed in running the first standalone binary created with MSYS2. This one is based on ldd.exe, just so this can be considered finished.

kayhayen commented 5 years ago

Released as 0.6.2

zoolyka commented 5 years ago

I still got this Error, cannot find 'python37m.lib' file. error using latest MSYS2 and Nuitka 0.6.5. Also installed the above packages. Where can I find this file and where to put it?