Closed ogrisel closed 8 years ago
ogrisel
commented
8 years ago Ok apparently this is just a consequence of that the MSVCR100 linkage is not implemented yet. If I try to build Cython using Python 2.7.11 and the matching mingwpy distribution it works as expected.
Edit: This is a wrong explanation.
carlkl
commented
8 years ago @ogrisel, the mingwpy wheel copies libpython34.dll.a as well as libmsvcr100.a to
ogrisel
commented
8 years ago The error messages above were obtained by using gcc installed by the mingwpy wheel in a dedicated venv.
ogrisel
commented
8 years ago Here is the content of the libs folder of the mingwpy venv (Python 3.4.4):
(mingwpy) PS C:\msys64\home\ogrisel\venv\mingwpy\libs> ls
Directory: C:\msys64\home\ogrisel\venv\mingwpy\libs
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 1/21/2016 10:46 PM 1272484 libmsvcr100.a
-a---- 1/22/2016 3:43 PM 1046676 libpython34.a
-a---- 1/21/2016 10:46 PM 979468 libpython34.dll.a
-a---- 1/22/2016 3:43 PM 27962 python34.defpython34.def is the list of exported symbols from libpython34.dll. This file is not part of python-3.4.4.msi neither part of mingwpy, however, this file doesn't harm. libpython34.a isn't part of mingwpy either. mingwpy contains two import libraries: libpython34.dll.a AND libmsvcr100.a.
Now the tricky parts:
Please use the import libraries contained in the mingwpy wheel!
I guess I have to create a script to validate and re-create (if needed) correct import libraries.
BTW: Cython-0.23b1 compiled out of the box with mingwpy: see https://anaconda.org/carlkl/cython
ogrisel
commented
8 years ago I must have made a mistake when installing the wheel? Let me try again. That is weird though. I don't remember having done any else than just creating a venv and installing the wheel with pip.
ogrisel
commented
8 years ago I just created a fresh virtualenv for python 3.4 (64 bit) and installed mingwpy in it:
(py34) PS C:\msys64\home\ogrisel\cython> pip show mingwpy
---
Metadata-Version: 2.0
Name: mingwpy
Version: 0.1.0b3
Summary: mingwpy - the python friendly windows compiler toolchain
Home-page: mingwpy.org
Author: mingwpy organization
Author-email: cmkleffner@gmail.com
License: MIT
Location: c:\users\ogrisel\py34\lib\site-packages
Requires:the content of the libs folder looks correct now:
(py34) PS C:\msys64\home\ogrisel\cython> ls C:\Users\ogrisel\py34\libs
Directory: C:\Users\ogrisel\py34\libs
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 1/23/2016 10:47 AM 1272484 libmsvcr100.a
-a---- 1/23/2016 10:47 AM 979468 libpython34.dll.a
However I still get the same errors when building cython (this time I tried from the 0.23b2 tag of the git repo after a git clean -xdf):
(py34) PS C:\msys64\home\ogrisel\cython> python .\setup.py build --compiler=mingw32 install
Unable to find pgen, not compiling formal grammar.
running build
running build_py
creating build
creating build\lib.win-amd64-3.4
copying cython.py -> build\lib.win-amd64-3.4
creating build\lib.win-amd64-3.4\Cython
copying Cython\CodeWriter.py -> build\lib.win-amd64-3.4\Cython
copying Cython\Coverage.py -> build\lib.win-amd64-3.4\Cython
copying Cython\Debugging.py -> build\lib.win-amd64-3.4\Cython
copying Cython\Shadow.py -> build\lib.win-amd64-3.4\Cython
[...]
running build_ext
cythoning C:\msys64\home\ogrisel\cython\Cython\Plex\Scanners.py to C:\msys64\home\ogrisel\cython\Cython\Plex\Scanners.c
building 'Cython.Plex.Scanners' extension
C:\Users\ogrisel\py34/Scripts\gcc.exe -mdll -O -Wall -Ic:\python34\include -Ic:\python34\include -c C:\msys64\home\ogris
el\cython\Cython\Plex\Scanners.c -o c:\msys64\home\ogrisel\cython\cython\plex\scanners.o
writing c:\msys64\home\ogrisel\cython\cython\plex\Scanners.def
C:\Users\ogrisel\py34/Scripts\gcc.exe -shared -s c:\msys64\home\ogrisel\cython\cython\plex\scanners.o c:\msys64\home\ogr
isel\cython\cython\plex\Scanners.def -Lc:\python34\Libs -LC:\Users\ogrisel\py34\libs -LC:\Users\ogrisel\py34\PCbuild\amd
64 -lpython34 -lmsvcr100 -o build\lib.win-amd64-3.4\Cython\Plex\Scanners.pyd
c:\msys64\home\ogrisel\cython\cython\plex\scanners.o:Scanners.c:(.text+0x3): undefined reference to `__imp__Py_NoneStruct'
c:\msys64\home\ogrisel\cython\cython\plex\scanners.o:Scanners.c:(.text+0x110): undefined reference to `__imp__Py_NoneStruct'
c:\msys64\home\ogrisel\cython\cython\plex\scanners.o:Scanners.c:(.text+0x13e): undefined reference to `__imp__Py_NoneStruct'
[...]Ok cython builds correctly if I install mingwpy directly at the top level python with c:\Python34\Scripts\pip instead of using a venv.
This is weird because I did not have the issue with Python 2.7 where I could build cython with mingwpy only installed in a venv instead of the top level Python.
ogrisel
commented
8 years ago My virtualenv version is up-to-date: 14.0.1.
ogrisel
commented
8 years ago Here is the compile command without venv that works:
C:\Python34\Scripts\gcc.exe -shared -s c:\msys64\home\ogrisel\cython\cython\plex\scanners.o c:\msys64\home\ogrisel\cython\cython\plex\Scanners.def -LC:\Python34\libs -LC:\Python34\PCbuild\amd64 -lpython34 -lmsvcr100 -o build\lib.win-amd64-3.4\Cython\Plex\Scanners.pydhere is the command in the venv when it fails:
C:\Users\ogrisel\py34/Scripts\gcc.exe -shared -s c:\msys64\home\ogrisel\cython\cython\plex\scanners.o c:\msys64\home\ogrisel\cython\cython\plex\Scanners.def -Lc:\python34\Libs -LC:\Users\ogrisel\py34\libs -LC:\Users\ogrisel\py34\PCbuild\amd64 -lpython34 -lmsvcr100 -o build\lib.win-amd64-3.4\Cython\Plex\Scanners.pydNote that both PCbuild paths do not exist on this box so this is not the cause of the issue.
carlkl
commented
8 years ago the venv case tries 2 different folders to resolve the locations of import libraries:
-Lc:\python34\Libs -LC:\Users\ogrisel\py34\libs (with venv)
-LC:\Python34\libs (without)
I never tried virtualenv (I should do), but I think this is a bug in virtualenv.
If mingwpy is located in C:\Users\ogrisel\py34 (not C:\Python34) -LC:\Python34\libs shouldn't be used in the link command. Can you check if the import libraries in both folders are correct?
ogrisel
commented
8 years ago Under Python 2.7, when I build cython from a venv I get the following build command (that works):
(py27) PS C:\msys64\home\ogrisel\cython> C:\Users\ogrisel\py27/Scripts\gcc.exe -shared -s c:\msys64\home\ogrisel\cython\cython\plex\scanners.o c:\msys64\home\ogrisel\cython\cython\plex\Scanners.def -Lc:\python27\Libs -LC:\Users\ogrisel\py27\libs -LC:\Users\ogrisel\py27\PCbuild\amd64 -LC:\Users\ogrisel\py27\PC\VS9.0\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\Cython\Plex\Scanners.pydThe folders C:\Users\ogrisel\py27\PCbuild\amd64 and C:\Users\ogrisel\py27\PC\VS9.0\amd64 do not exist there either (actually neither do C:\Users\ogrisel\py27\PCbuild nor C:\Users\ogrisel\py27\PC).
Here is the content of the 2 libs folder when using the py27 venv, first the top level python libs folder:
(py27) PS C:\> ls C:\python27\Libs
Directory: C:\python27\Libs
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 12/5/2015 8:42 PM 1648 bz2.lib
-a---- 12/5/2015 8:45 PM 826618 libpython27.a
-a---- 12/5/2015 8:41 PM 1716 pyexpat.lib
-a---- 12/5/2015 8:40 PM 229328 python27.lib
-a---- 12/5/2015 8:41 PM 1700 select.lib
-a---- 12/5/2015 8:42 PM 1784 unicodedata.lib
-a---- 12/5/2015 8:42 PM 1734 winsound.lib
-a---- 12/5/2015 8:45 PM 1884 _bsddb.lib
-a---- 12/5/2015 8:41 PM 1716 _ctypes.lib
-a---- 12/5/2015 8:41 PM 21568 _ctypes_test.lib
-a---- 12/5/2015 8:41 PM 1880 _elementtree.lib
-a---- 12/5/2015 8:43 PM 1734 _hashlib.lib
-a---- 12/5/2015 8:41 PM 1666 _msi.lib
-a---- 12/5/2015 8:41 PM 1952 _multiprocessing.lib
-a---- 12/5/2015 8:42 PM 1916 _socket.lib
-a---- 12/5/2015 8:42 PM 1734 _sqlite3.lib
-a---- 12/5/2015 8:42 PM 1666 _ssl.lib
-a---- 12/5/2015 8:41 PM 1750 _testcapi.lib
-a---- 12/5/2015 8:43 PM 1734 _tkinter.liband the libs folder form the venv itself:
(py27) PS C:\> ls C:\Users\ogrisel\py27\libs
Directory: C:\Users\ogrisel\py27\libs
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 1/23/2016 11:41 AM 1096598 libmsvcr90.a
-a---- 1/23/2016 11:41 AM 790036 libpython27.dll.aso it looks very similar to the Python 3.4 case, I cannot spot the difference.
ogrisel
commented
8 years ago I will reinstall a new Python 3.4 top level folder to create a new venv and do more investigations.
carlkl
commented
8 years ago I tried myself with python-3.4.4 (amd64) and virtualenv. mingwpy places its import files into the new environment.
-LC:\msys64\home\user\venv\py34\libs -Ld:\devel\py\python-3.4.4.amd64\Libs
I have no idea of a good workaround though.
ogrisel
commented
8 years ago Yes I just found out that switching the -L flags order fixes the problem on my end as well. Copying the import files to the global libs is the wrong fix, the whole point of virtualenv is to isolate them from one another to be able to test different versions of libraries (compiled or not) in parallel.
ogrisel
commented
8 years ago I don't understand why we don't observe the issue with Python 2.7. The -L flags order is the same as for Python 3.4 but gcc works just fine for Python 2.7 while it fails with Python 3.4.
carlkl
commented
8 years ago This is the kind of weirdness that made me headaches all the time in the last two years.
What happens during the linking step is the following:
-L<somepath1> -L<somepath2> -lpython34 tries to find in that order:
libpython34.dll.alibpython34.apython34.lib - yes, binutils also tries to support MS import files!Of course <somepath1> is used first. For python-2.7 there is possibly a libpython27.a inside the global lib folder that is going to be used, provided by CPython installer itself. For the python-3.4 case this file is missing, so binutils tries to use python34.lib instead of libpython34.dll.a in the venv folder. Renaming python34.lib to something proves that behavior. It is also possible, that in the case of python-2.7 python27.lib is consumable by binutils - I didn't checked that case.
workaround: If the pollution of the global libs folder is out of the scope I see only one possible solution:
patching virtualenv to switch the order of the libs files, at least in case of mingw usage - possibly in the monkeypatching code of virtualenv inside the env's site-package?
ogrisel
commented
8 years ago patching virtualenv to switch the order of the libs files, at least in case of mingw usage - possibly in the monkeypatching code of virtualenv inside the env's site-package?
you mean patching distutils?
ogrisel
commented
8 years ago I think you should bring the issue on the distutils sig mailing list. Maybe @njsmith has an opinion on this issue?
njsmith
commented
8 years ago IIUC, .lib and .a files are the same thing on windows, it's just that the windows-style and the mingw-w64-style toolchains have different traditions for which extension to use -- in principle we shouldn't even need to generate our own .lib files for python. (IIUC the main reason mingw-w64 generates its own .lib files for all the system libraries is because the provided .lib files have the wrong licensing, not the wrong format.)
I have some faint memory though of reading (maybe on the mingw-w64 mailing list?) that there was some update to the .lib/.a file format on Windows and binutils hasn't yet added support for the newest version... if so then that might explain why the python.org-provided .lib file for 2.7 works (b/c it's generated by the 2008 toolchain) while the python.org-provided .lib file for 3.4 doesn't (b/c it's generated by the 2010 toolchain, which maybe switched to the new format).
If this is correct then the right fix is to update binutils to support the new format, so that mingwpy could just use the real pythonXX.lib and stop worrying about all this junk :-). This would also be useful in case MS comes through and fixes the licensing on their SDK, and for general improvement of msvc/mingw-w64 interoperability. No idea how difficult this would be or if it's likely to happen but it would be good to at least touch base with upstream about it. (Assuming this is correct.)
If the situation is that there's an unsupported file format in ld and it won't be fixed immediately, then it might also make sense to add some workaround code to ld so if it finds a file in the unsupported format, it notices this and then keeps searching instead of giving up.
A really hacky workaround would be for the mingwpy .spec file to insist that the directory with its special libpythonXX.a always comes first in the linker search path :-)
carlkl
commented
8 years ago I guess this is a bug - Microsoft import libraries not correctly handled for x64 - in binutils as described here
I check it with the new toolchain build
njsmith
commented
8 years ago @carlkl: Some spelunking suggests that that fix may not be in the latest mingw-builds yet; if not though then I guess it should be trivial to backport...
carlkl
commented
8 years ago that was my first thought too ...
ogrisel
commented
8 years ago Great indeed the patch in Bugzilla seems very simple to backport in our branch of mingw-build (assuming this is the correct fix). As it has been recently merged in master we only need to maintain the backport for the 2.25 release of binutils and can drop it as soon as 2.26 is out.
ogrisel
commented
8 years ago Actually issue 16598 is probably not the same as our own issue as the patch described as a fix in the ticket is already applied by the mingw-build scripts:
njsmith
commented
8 years ago Excellent! Now we just need to check whether it actually works :-) On Jan 24, 2016 9:33 AM, "Olivier Grisel" notifications@github.com wrote:
Actually issue 16598 as the patch described as a fix in the ticket is already applied by the mingw-build scripts:
- https://github.com/mingwpy/mingw-builds/blob/mingwpy-dev/scripts/binutils.sh#L62
https://github.com/mingwpy/mingw-builds/blob/mingwpy-dev/patches/binutils/27aaeda.diff
— Reply to this email directly or view it on GitHub https://github.com/mingwpy/mingwpy.github.io/issues/31#issuecomment-174321735 .
ogrisel
commented
8 years ago No my previous comment states that the fix is already in the mingwpy version where we observed the problem in the first place. So this is not the fix for our issue unfortunately...
carlkl
commented
8 years ago It doesn't work as it is already part of mingwpy 0.1.0b3. There is another patch not included yet: fixes-a-problem-recognizing-libraries-created-by-VS.patch, that has to be tested.
njsmith
commented
8 years ago Doh :-( I thought you were testing the older mingwpy build still. On Jan 24, 2016 11:45 AM, "carlkl" notifications@github.com wrote:
It doesn't work as it is already part of mingwpy 0.1.0b3. There is another patch not included yet: fixes-a-problem-recognizing-libraries-created-by-VS.patch, that has to be tested.
— Reply to this email directly or view it on GitHub https://github.com/mingwpy/mingwpy.github.io/issues/31#issuecomment-174334329 .
njsmith
commented
8 years ago @ogrisel: Would it be possible to do a minimal clean check to confirm that the problem is that it works to link to upstream's python27.lib, or our libpython27.dll.a, or our libpython34.dll.a, but that attempting to link to upstream's python34.lib as does not work and produces the given error message referring to __imp__PyNoneStruct? (For example, you could copy those files each to their own directory, and then try rerunning the gcc command manually with -Lthat-directory.)
I'm hoping to have a simple description of the problem we can take to e.g. Kai, rather than just pointing them at this thread :-)
(And it occurs to me also that it would be very nice if we can convince binutils and msvc to use the same .lib files, because otherwise for any C libraries we package like openblas, we will have to distribute both .lib formats, which means that our build machine will need both toolchains, and it'll be pretty annoying all around.)
carlkl
commented
8 years ago @njsmith, I tried out to link libpython34.dll.a (works) libpython34.a (works) and python34.lib (fails) on amd64. I didn't make a cross check with win32, python27 etc as I'm running out of time today. Tomorow I check this and compare it with a rebuild of binutils with the fixes-a-problem-recognizing-libraries-created-by-VS.patch.
njsmith
commented
8 years ago For the record: the fixes-a-problem-recognizing-libraries-created-by-VS.patch appears to be the patch for this bug: https://sourceware.org/bugzilla/show_bug.cgi?id=17910 . The report there has a different error message unfortunately so it likely won't help, but fingers crossed...
njsmith
commented
8 years ago Asked about this on #mingw-w64 (on OFTC) and then got called away, but:
<njs> mingwandroid: hey, what's up :-)
<mingwandroid> njs: hey. not much, how are you?
<njs> anyone know why one would get an error like this when trying to use current mingw-w64/binutils to link a .dll against a msvc-generated .lib file?: c:\...\cython\plex\scanners.o:Scanners.c:(.text+0x3): undefined reference to `__imp__Py_NoneStruct'
<njs> (Py_NoneStruct should be exported from the .lib file -- it works with a MSVC 2008-generated .lib file or a mingw-w64-generated .lib file, but not with a MSVC 2010-generated .lib file.)
<njs> mingwandroid: doing okay :-)
<jon_y> use nm to take a look?
<njs> oops, someone at the door
<jon_y> party van here to pick you up
<mingwandroid> njs: which architectures are you trying here?
<mingwandroid> njs: I must sleep, but here's a gist that seems to work for me on MSYS2 MinGW-w64 GCC and https://www.python.org/ftp/python/3.4.4/python-3.4.4.amd64.msi (it's very raw, not using cython, or even python's module extension build-system!)
<mingwandroid> njs: https://gist.github.com/mingwandroid/7bba87e614f6c954f604so mingwandroid (= Ray Donnelly) is apparently able to successfully link a trivial module using the python.org-provided python34.lib together with msys2's gcc, which may help narrow things down...
ogrisel
commented
8 years ago @njsmith I can build an extension against Python 3.4 when mingwpy is installed in the top level Python. The problem only occurs when installing mingwpy in a venv: in that case we have 2 -L flags in the gcc command line and this only causes the problem (see https://github.com/mingwpy/mingwpy.github.io/issues/31#issuecomment-174212330 and the following comments).
The top level libs folder has the libpython34.a file from Python, and the venv libs folder has the mingwpy provided libpython34.a.dll (along with libmsvcr100.a). If the ordering of the -L flags is switched such that gcc can lookup libpython34.a.dll first then the build works.
carlkl
commented
8 years ago That is what I found yesterday, see https://github.com/mingwpy/mingwpy.github.io/issues/31#issuecomment-174212330. The question is why the MSVC import library doesn't work properly. It seems to work with newer toolchains, presumably as they are equipped with binutils-2.2.5 (not binutils-2.2.4 as in gcc-4.9.x based toolchains). The applied patches to binutils are almost the same.
It would be good to make this works. But I'm not sure if VS 2015 import libraries are already supported by binutils. There is a new tool in mingw-w64 called genlib to produce MSVC compatible import libraries. This tool can be used in favour of dlltool. But this is not part of binutils.
The safest approach is to use import libaries build by the tools of the toolchain. So the safest workaround IMHO is to switch the ordering of the -L flags.
ogrisel
commented
8 years ago So the safest workaround IMHO is to switch the ordering of the -L flags.
But this would mean that extension module maintainers would have to use an alternative implementation of distutils.cywincompiler.Mingw32CCompiler that has to be registered in the setup.py file of their project. This is annoying because it would require them to edit their source code just to support windows.
I wonder if we cannot find a less invasive way to do it.
njsmith
commented
8 years ago @ogrisel: Right, the issues with the different directories are a distraction; the point is that it shouldn't matter which import library ld finds first, because both import libraries should work identically :-). Maybe it's not realistic to make that happen in real life, but if it can be done then it's definitely the simplest solution, because then we don't need to keep track of all these things.
In general it definitely is supposed to work that way... there are bug reports in the binutils bugzilla (linked above) of people saying "hey this used to work, but it broke" and then after the patch it works again. And VS 2015 can't have anything to do with it, because this is 3.4, which is using VS 2010...
ogrisel
commented
8 years ago I agree.
njsmith
commented
8 years ago One thing to try would be to see if the mingwpy gcc gets the same result as the msys2 gcc does here: https://gist.github.com/mingwandroid/7bba87e614f6c954f604 , linking using the python.org-distributed import library. If it fails, then there's something msys2 has the mingwpy is missing; if the simple test case in the gist works, then there's something more complicated going wrong with the import library and that particular .c file than just "binutils doesn't understand this file".
ogrisel
commented
8 years ago I get the same result with mingwpy 0.1.0b3 gcc:
(py34) PS C:\src> gcc -shared -LC:\Python34_x64\libs -lpython34 -c test.c -o test.dll
(py34) PS C:\src> nm .\test.dll
0000000000000000 b .bss
0000000000000000 d .data
0000000000000000 p .pdata
0000000000000000 r .rdata$zzz
0000000000000000 t .text
0000000000000000 r .xdata
U __imp__Py_NoneStruct
0000000000000000 T DLLMainI checked that the gcc in my PATH is the one installed by mingwpy in my py34 venv:
(py34) PS C:\src> get-command gcc
CommandType Name Version Source
----------- ---- ------- ------
Application gcc.exe 0.0.0.0 C:\venv\py34/Scripts\gcc.exeLinking https://gist.github.com/mingwandroid/7bba87e614f6c954f604 works at a first sight, but it produces an improper test.dll. This dll created isn't a valid PE executable, nor it is linked against python34.dll. This testcase is too simple, I or @njsmith should produce another one.
For the cython case I tried now several executables for ld.exe and collect2.exe (cherrypicking from mingw-builds, gcc-4.9.2, 4.9.3, 5.3.0) and always runs into the same error trying to link python34.lib.
It seems, that binutils has problems finding symbols from the data section. nm isn't able to find 187 symbols from python34.lib but those symbols are exported in libpython34.dll.a
__imp_PyBaseObject_Type
__imp_PyBool_Type
...
__imp_PyExc_ArithmeticError
__imp_PyExc_AssertionError
..
__imp_PyExc_RecursionErrorInst
__imp_PyExc_ReferenceError
__imp_PyExc_ResourceWarning
__imp_PyExc_RuntimeError
__imp_PyExc_RuntimeWarning
..
__imp_PyFilter_Type
__imp_PyFloat_Type
__imp_PyFrame_Type
__imp_PyFrozenSet_Type
__imp_PyFunction_Type
__imp_PyGen_Type
__imp_PyGetSetDescr_Type
__imp_PyImport_FrozenModules
__imp_PyImport_Inittab
..
__imp__Py_NoneStruct
__imp__Py_NotImplementedStruct
__imp__Py_PackageContext
__imp__Py_SwappedOp
__imp__Py_TrueStruct
__imp__Py_ascii_whitespace
__imp__Py_ctype_table
__imp__Py_ctype_tolower
__imp__Py_ctype_toupperEDIT: explicit linking against python34.dll technically works (without using import libraries). Maybe the problem lies in the MSVC import library python34.lib not in binutils?
carlkl
commented
8 years ago My impression was always, that linking against MSVC generated import libraries should work in theory but fails in practice. The usage of import libraries generated by the mingwpy toolchain (gendef/dlltool) is preferable and the recommened approach. The simplest workaround ist to create a python import library (libpython34.dll.a i.e.) in the global libs folder as well. This step will be only necessary if virtualenv is used.
ogrisel
commented
8 years ago The simplest workaround ist to create a python import library (libpython34.dll.a i.e.) in the global libs folder as well. This step will be only necessary if virtualenv is used.
The think is that wheel cannot (and should not try to) install any file outside of the venv folder: the user might have the permissions to install files in their venv but not in the top level Python install folder (which might be a folder only writeable by the Administrator user of the system).
This is the whole point of isolation with venvs.
njsmith
commented
8 years ago I don't understand what you mean about the gist not producing a valid PE file (it's a senseless one for sure but it should be a valid dll file, surely?). If that .dll ends up without a link to python34.dll though then potentially that's the problem right there? I mean that command ought to produce a .dll that's linked against python34.dll, surely.
carlkl
commented
8 years ago > file test.dll
test.dll: data
> ldd test.dll
ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x772d0000)
kernel32.dll => /c/WINDOWS/system32/kernel32.dll (0x770b0000)
KERNELBASE.dll => /c/WINDOWS/system32/KERNELBASE.dll (0x7fefd170000)
depends (MS tool dependecy walker):
Error: At least one file was not a 32-bit or 64-bit Windows module.
> objdump -D test.dll
test.dll: file format pe-x86-64
Disassembly of section .text:
0000000000000000 <DLLMain>:
0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: 48 83 ec 10 sub $0x10,%rsp
8: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # f <DLLMain+0xf>
f: 48 89 45 f8 mov %rax,-0x8(%rbp)
13: b8 00 00 00 00 mov $0x0,%eax
18: 48 83 c4 10 add $0x10,%rsp
1c: 5d pop %rbp
1d: c3 retq
1e: 90 nop
1f: 90 nop
Disassembly of section .xdata:
0000000000000000 <.xdata>:
0: 01 08 add %ecx,(%rax)
2: 03 05 08 12 04 03 add 0x3041208(%rip),%eax # 3041210 <.xdata+0x3041210>
8: 01 50 00 add %edx,0x0(%rax)
...
Disassembly of section .pdata:
0000000000000000 <.pdata>:
0: 00 00 add %al,(%rax)
2: 00 00 add %al,(%rax)
4: 1e (bad)
5: 00 00 add %al,(%rax)
7: 00 00 add %al,(%rax)
9: 00 00 add %al,(%rax)
...
Disassembly of section .rdata$zzz:
0000000000000000 <.rdata$zzz>:
0: 47 rex.RXB
1: 43 rex.XB
2: 43 3a 20 rex.XB cmp (%r8),%spl
5: 28 78 38 sub %bh,0x38(%rax)
8: 36 5f ss pop %rdi
a: 36 34 2d ss xor $0x2d,%al
d: 77 69 ja 78 <.rdata$zzz+0x78>
f: 6e outsb %ds:(%rsi),(%dx)
10: 33 32 xor (%rdx),%esi
12: 2d 73 65 68 2d sub $0x2d686573,%eax
17: 72 65 jb 7e <.rdata$zzz+0x7e>
19: 76 32 jbe 4d <.rdata$zzz+0x4d>
1b: 30 31 xor %dh,(%rcx)
1d: 35 30 36 2c 20 xor $0x202c3630,%eax
22: 6d insl (%dx),%es:(%rdi)
23: 69 6e 67 77 70 79 20 imul $0x20797077,0x67(%rsi),%ebp
2a: 62 (bad)
2b: 75 69 jne 96 <.rdata$zzz+0x96>
2d: 6c insb (%dx),%es:(%rdi)
2e: 64 29 20 sub %esp,%fs:(%rax)
31: 34 2e xor $0x2e,%al
33: 39 2e cmp %ebp,(%rsi)
35: 32 00 xor (%rax),%al
...Huh, I wonder if it got statically linked or what...
Cc: @mingwandroid
mingwandroid
commented
8 years ago I may have messed my flags up as I was on my way to bed. I'll try to check later. If someone has good reproduction steps to help get me up to speed that would help though. Soon I'll be devoting much more time to windows and python, both professionally and spare.
carlkl
commented
8 years ago I reproduced the test again (https://gist.github.com/mingwandroid/7bba87e614f6c954f604#gistcomment-1680099) and got a valid DLL binary linked against python34.dll and msvcr100.dll.
And again: the link step works using:
libpython34.alibpython34.dll.apython34.dlland it fails with test.c:(.text+0xb): undefined reference toimpPy_NoneStruct'using the MS import librarypython34.lib`
All the symbols out of the data section (marked as DATA in the gendef produced def file) can't be resolved using the original MS import library.
@mingwandroid, do you know a workaround for that behavior?
I get the following errors (truncated):
Full log at: https://gist.github.com/ogrisel/ba513e2aa19583202973