Add /NODEFAULTLIB:MSVCRT to _msvccompiler

[zooba]

BPO	24872
Nosy	@malemburg, @pfmoore, @larryhastings, @tjguk, @zware, @zooba, @mikofski

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/zooba' closed_at = created_at = labels = ['extension-modules', 'type-bug', 'OS-windows', 'docs'] title = 'Add /NODEFAULTLIB:MSVCRT to _msvccompiler' updated_at = user = 'https://github.com/zooba' ``` bugs.python.org fields: ```python activity = actor = 'bwanamarko' assignee = 'steve.dower' closed = True closed_date = closer = 'steve.dower' components = ['Documentation', 'Extension Modules', 'Windows'] creation = creator = 'steve.dower' dependencies = [] files = [] hgrepos = [] issue_num = 24872 keywords = [] message_count = 27.0 messages = ['248646', '248651', '248652', '248654', '248676', '248681', '248687', '248688', '248689', '248690', '248691', '248715', '248716', '248722', '248724', '248725', '248736', '248744', '248747', '249041', '249059', '249589', '249591', '249593', '249596', '249615', '260581'] nosy_count = 8.0 nosy_names = ['lemburg', 'paul.moore', 'larry', 'tim.golden', 'cgohlke', 'zach.ware', 'steve.dower', 'bwanamarko'] pr_nums = [] priority = 'high' resolution = 'out of date' stage = None status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue24872' versions = ['Python 3.5', 'Python 3.6'] ```

[zooba]

The C Runtime used for Python 3.5 and later consists of two DLLs. One, ucrtbase.dll, is an operating system component that can be assumed to be available and up to date (or will be installed by the Python installer).

The second DLL is vcruntimeXXX.dll, where XXX is a version number that is directly tied to the version of the compiler that was used to build.

In order to maintain version-independence from the C Runtime, vcruntime should be statically, rather than dynamically, linked. This ensures that extensions linking to a different vcruntime will run on machines with builds of Python prior to that version being available (i.e. we'd have needed a time machine to be able to ensure all future versions of vcruntime are available). This would essentially put us back in a place where you need to match compilers to build extensions.

To achieve this semi-static linking, the following options must be used.

cl /c /MT /GL \<file.c> link \<file.obj> /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib

Building with /MT links everything statically, and the last two linker options substitute the non-static ucrtbase.dll. /GL and /LTCG (link-time code generation) ensures the correct stubs for dynamic linking are created - compiling with /MD instead of /MT generates these at compile time instead of link time.

One problem appears to be including custom-built static libraries that compiled objects with /MD instead of /MT:

cl /C testlib.c /MD lib testlib.c cl /C test.c /MT /GL link test.obj testlib.lib /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib \<linker errors occur>

These are because testlib.lib pulls in the MSVCRT library (which indirectly includes the dynamic vcruntime.lib and ucrt.lib) instead of the LIBCMT library (which indirectly includes the static libvcruntime.lib and libucrt.lib). Building test.c with /MT pulls in LIBCMT and conflicts ensue.

This may be fixed by using /MD for test.c, but this causes a dependency on vcruntimeXXX.dll. It appears that it can also be fixed by excluding MSVCRT from the final link step:

link test.obj testlib.lib /LTCG **/NODEFAULTLIB:msvcrt.lib** /NODEFAULTLIB:libucrt.lib ucrt.lib

Christoph - I know you've already patched your _msvccompiler.py to use /MD instead of /MT, but I wonder if you could try reverting that and adding the /NODEFAULTLIB:msvcrt.lib linker option instead and see if that helps? For my test cases it's been fine, but you have a much larger collection of libraries to work with. I'm very keen to see if this is a satisfactory solution.

FWIW, I've spent a lot of time considering ways we could actually include vcruntime*.dll such that old Python versions would include or get future versions of the DLL and have come up with nothing. Requiring users to install every VC redistributable that is released is unacceptable to me, and I would much rather help extension builders to create extensions that do not require extra DLLs.

[50363613-ac32-4562-87b1-ef078e34ece0]

Thank you for looking into this.

I just tried '/NODEFAULTLIB:msvcrt.lib' with Pillow and matplotlib but still get the linker errors:

tiff.lib(jbig.obj) : error LNK2001: unresolved external symbol __imp_memchr

freetype.lib(ftbase.obj) : error LNK2001: unresolved external symbol __impstrrchr freetype.lib(truetype.obj) : error LNK2001: unresolved external symbol \_impstrstr freetype.lib(type1.obj) : error LNK2001: unresolved external symbol \_impmemchr freetype.lib(sfnt.obj) : error LNK2001: unresolved external symbol \_imp_memchr

Looks like I'll have to rebuild all the 3rd party libraries with '/MT' and link DLLs with '/NODEFAULTLIB:libucrt.lib ucrt.lib' to remove the dependency on vcruntimeXXX.dll.

This change should probably be mentioned in the release notes.

Maybe also bring it to the attention of Ilan Schnell (ilan) and Cournapeau David (cdavid) who build packages for Anaconda and Canopy.

[malemburg]

What effect does this static linking of the VC runtime have on libraries that extension modules link at dynamically ?

E.g. say I have an extension which links against an ODBC driver DLL using a different VC runtime than the one used to build Python.

In the past, the has never been an issue, so adding a requirement to rebuild external libraries is pretty much a no-go for Python extension writing. You often have the case where you cannot rebuild the external libraries you want to link against.

[zooba]

I guess those imports are expected to come from vcruntime. Rebuilding the static libraries may be the only option then, sorry (on the bright side, VC 14 has much better C99 support than earlier versions, so things should be better once we get past the problems caused by change).

Marc-Andre: there are a few concerns with including DLLs that aren't new with any of the 3.5 changes.

• depending on another CRT version is fine *if* it is available (users may have to be told/helped to install the redistributable themselves)
• CRT state will not be shared between versions. This is most obviously a problem if file descriptors are shared, but may also appear when writing console output, doing floating-point calculations, threading, and memory management.
• potentially many more issues if C++ is used, but since Python doesn't use C++ this is mainly a concern where you have two DLLs using C++ and different runtimes (the CRT is partially/fully implemented in C++, so issues may theoretically occur with only one DLL using C++, but I'm yet to see any in practice or even identify any specific issues - maybe it's fine? I'm not going to guarantee it myself)

The parts of the new CRT that will be statically linked are (broadly):

• process/DLL initialization
• exception handling
• some functions that are also compiler intrinsics

The first two don't apply for C static libraries, and even seem to be okay if the static lib is the one declaring main(). I did make the argument that they should be statically linked by default, but the response was that security patches may need to be applied to those parts of the code (particularly exception handling - corrupting the stack and then raising an exception is a great way to jump to whatever code you like). If Python shipped this DLL, it would not be updateable anyway, so static linking is a considered risk that could cause us to someday ship a security update due to a change in the CRT (exactly as has been the case for every other version of Python - nothing new here).

The last item is what Christoph just hit - because the compiler may partially inline those functions (memchr, strchr, etc.) for fast paths, it needs intimate knowledge about how they are implemented. As a result, they go in the compiler-version specific DLL and not the generic one. This lets the implementations make assumptions about what conditions the compiler has already tested and/or proven, but since this could change with compiler version the implementation may also change.

Since the functions are stateless, there's no issue if they happen to be included in a final DLL multiple times for different versions, but that will only happen if the original library is compiled with /MT (I checked - there's no way to override the code generated here under /MD other than to change the compiler option, but link-time code generation should be able to take /MT-compiled libraries and dynamically link them).

So basically, there are no new restrictions on linking to dynamic libraries. The biggest concern would be if those libraries depend on msvcr100.dll and worked with Python 3.4 (which shipped msvcr100), they may not work on Python 3.5 unless the user has obtained msvcr100.dll some other way.

[50363613-ac32-4562-87b1-ef078e34ece0]

FWIW, I rebuilt static libraries for zlib, jbig, jpeg, openjpeg, tiff, webp, lcms, and freetype with /MT flag (a tedious task) and was able to build matplotlib and Pillow using Python 3.5.0rc1. As expected there is no dependency on the vcruntime DLL.

Even if everything is built with /MT and using UCRT, some extensions will still depend on version specific Visual C runtime DLLs, e.g. OpenMP, requiring users or distributors to install/bundle the VC runtime package.

[zooba]

I expect that, but most extensions don't seem to be in that category so this will help remove the administrator barrier.

Thanks for going through that tedious process. I'll put up a patch later today and submit for 3.5.0.

[zooba]

Actually, on rereading this (during daylight hours, previous response was before 6am my time) the patch wouldn't help. I need to write some porting notes for rebuilding static libraries with suitable settings. I'll base it on my initial post and find somewhere appropriate in the docs (and maybe blog about it too for SEO purposes).

[50363613-ac32-4562-87b1-ef078e34ece0]

Thanks for going through that tedious process \~140 libraries to go.

I hit the wall last night trying to build Boost DLLs. Boost's build tool b2 does not allow link=shared runtime-link=static, hence the /MT /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib magic does not work.

[zooba]

Boost requires C++ anyway doesn't it? So the full redist will be required. These options are only useful for pure C sources.

[50363613-ac32-4562-87b1-ef078e34ece0]

Matplotlib and my own extensions are using C++ sources but do not depend on msvcp140.dll, just the ucrt. Am I missing something?

[zooba]

Probably I'm missing something. Maybe there's a subset of C++ that doesn't rely on it - a decent amount of the standard template library is generated at compile time.

If the dependency isn't there, it'll be fine.

Do you think it'll be worth having a check box in the installer to get the full runtime? Bearing in mind that most won't need it and many won't be able to install it? I'm not sure it is, but you've got a better idea of which packages are popular and which ones need it.

[50363613-ac32-4562-87b1-ef078e34ece0]

The matplotlib extensions compiled with Python 3.5.0rc1 (/MT) are larger than those compiled with 3.5.0b4 (/MD). The C++ runtime is statically linked. This seems undesirable for the same reasons the UCRT is not linked statically.

In "Introducing the Universal CRT" [1] James McNellis "strongly recommend against static linking of the Visual C++ libraries, for both performance and serviceability reasons". In "Visual Studio 2015 RTM Now Available" [2] the same author commented that one "may deploy the Universal CRT app-locally". Do these comments not apply to CPython?

[1] \http://blogs.msdn.com/b/vcblog/archive/2015/03/03/introducing-the-universal-crt.aspx\ [2] \http://blogs.msdn.com/b/vcblog/archive/2015/07/20/visual-studio-2015-rtm-now-available.aspx\

[50363613-ac32-4562-87b1-ef078e34ece0]

Another /MT only issue: cryptography-1.0 and other libraries depending on openssl fail to link to static MT openssl-1.0.1p:

cryptlib.obj : error LNK2001: unresolved external symbol __iob_func

This can be fixed manually [1].

[1] \http://openssl.6102.n7.nabble.com/Compiling-OpenSSl-Project-with-Visual-Studio-2015-td59416.html\

[malemburg]

On 15.08.2015 22:41, Steve Dower wrote:

Marc-Andre: there are a few concerns with including DLLs that aren't new with any of the 3.5 changes.

• depending on another CRT version is fine *if* it is available (users may have to be told/helped to install the redistributable themselves)
• CRT state will not be shared between versions. This is most obviously a problem if file descriptors are shared, but may also appear when writing console output, doing floating-point calculations, threading, and memory management.
• potentially many more issues if C++ is used, but since Python doesn't use C++ this is mainly a concern where you have two DLLs using C++ and different runtimes (the CRT is partially/fully implemented in C++, so issues may theoretically occur with only one DLL using C++, but I'm yet to see any in practice or even identify any specific issues - maybe it's fine? I'm not going to guarantee it myself)

These issues have always existed in the past, but were never a real problem, AFAIK, since the libraries intended to be used externally will typically come with e.g. memory management APIs to make sure they retain ownership of the allocated memory on their heap.

It is quite natural to have to run VCredist as part of an application installer to make sure that the target system has the right VC runtime DLLs installed (and the installer will do the checking).

The purpose of having DLLs for the runtime is to reduce overall size of the components as well as being able to easily address bugs and security issues in the runtime DLLs *without* having to recompile and redeploy all components using them.

By forcing or even suggesting statically compiled Python C extensions, we would break this goal and potentially put our users at risk.

IMO, we should follow the MS recommendations for "Deployment in Visual C++" as we did in the past:

https://msdn.microsoft.com/en-us/library/dd293574.aspx

""" You can statically link a Visual C++ library to an application—that is, compile it into the application—so that you don't have to deploy the Visual C++ library files separately. However, we caution against this approach because statically linked libraries cannot be updated in place. If you use static linking and you want to update a linked library, you have to recompile and redeploy your application. """

Perhaps I'm missing something, but if the only advantage of statically compiling in the runtime is to have users not need to run VCredist at install time, it's not worth all the added trouble this introduces.

If you are trying to make it possible to compile extensions with compilers following VC2015, then I also don't think this approach will work: the new compilers will use a new runtime and so issues you describe above come into play between the extensions and the interpreter.

In that scenario, they will create real problems, as far as I understand, the since the Python C API expects to be able to e.g. share FDs, internal state such as which locale to assume, or use and free memory allocated by either the interpreter or the extension in the resp. other component (e.g. PyArg_ParseTuple()).

So in the end, you'll still have to use the same compiler for extensions as the one used for compiling CPython to make sure you don't run into these issues - which is essentially the same situation as for Python \<=3.4.

[zooba]

Unfortunately applocal deployment doesn't solve the versioning issue - you'll always need to use VS 2015 to build for Python 3.5.

There are only a few more libraries that are affected by /MT. msvcp140.dll is the most likely one here. Doing the same /nodefaultlib dance for that might work, but it may not.

All of the shared state like memory allocation and file descriptors are in ucrt, which will not change with compiler versions. CPython itself does not require any extensions to use VS 2015 as long as they link to ucrtbase and not another CRT, but it's looking like the same build process won't work everywhere else.

Installing the full runtime requires administrative privileges, and removing that requirement was one of the priorities. If certain extensions require it or if distros install it that's fine, but I don't want distutils to *help* people make wheels that won't work on another machine because the runtime isn't there.

Depending on the size difference, statically linking the C++ parts may not be so bad. It's certainly no worse than including it local to Python, and all of its potentially shared state should be handled by the ucrt already.

Also, I've already had that OpenSSL issue fixed upstream. You must have an older version - I've been building it fine without patching for months now.

[malemburg]

If I understand you correctly, the only advantage of using /MT is not require admin privileges for installation of the VC2015 runtime libs.

Since VC2015 will be used by a lot of applications in a few months, and it's likely that MS will ship the runtime as Windows update anyway, the advantage seems minor.

OTOH, the requirement of linking against external libraries which you cannot recompile or don't support /MT is rather common and won't go away. And the need for security updates to the ucrt is rather inevitable as well based on experience with previous CRTs.

Being able to build a statically linked Python binary is a nice feature for some special application settings, but given the rather weak arguments for making this the default, I'm not convinced that this is a good way forward, esp. not when even MS itself recommends against doing this.

We can have Python run VCredist during the installation to make sure the runtime DLLs are available. Then no one will have a problem.

[zooba]

We can't have Python run VCredist because it requires administrative privileges and installing Python does not require those. This is actually one of the biggest complaints about the current installer, but because it looks like the design and not a bug it only comes up when you ask people directly what they'd like to change. (The most common bug is pip failing to install/uninstall, which is what you see on the issue tracker.)

The other benefits are that PYDs compiled with VS 2016+ (or whatever it's called) with these options will work with stock Python 3.5 on a clean machine - so if your extension links to python3.dll you can build once for all Python versions 3.5 and later and know that it will just copy and run. If you're building from source and only have a newer compiler than VC14, you will be able to build successfully. Also, we could decide to build some future Python 3.5.x with a newer compiler as it should not break any existing extensions (after much discussion and testing, obviously, but it is only feasible if we properly hide the VC version now).

To achieve these (apart from the last point) with VCredist, we need to be installing the current and all future versions of the redist with Python. Hence a time machine is required, and I haven't come up with any good way to simulate a time machine here.

not when even MS itself recommends against doing this.

I have separate advice (also from MS, from the same people who have been quoted previously, but in private conversations so I can't link to it) that if we want any chance of our plugin architecture being VC version independent, this is what we have to do. I'm emailing again to get more specific advice, but the official guidance has always been biased by wanting people to get the latest tools (which is why VC9 disappeared until my team made the case that sometimes people can't upgrade). We're still pushing hard to make this an acknowledged use case, and don't be surprised if at some point in the future official advice says "if you allow plugins, do what CPython did to help your users and developers".

The "/MT" == "statically linked" equivalence is an oversimplification in the presence of link-time code generation ("/GL" and "/LTCG" options), as we can take .obj or .lib files compiled with /MT and still use dynamic linking. The difference is we have to do it explicitly, which is what the "/nodefaultlib:libucrt.lib ucrt.lib" options do. If we add concrt140, msvcp140 and vccorlib140 to that as well (and probably the rest of the versions redistributables) then all of them will continue to be dynamically linked.

So basically, all the existing static libs could be built with /MT and still have their dependencies dynamically linked if that's what the final linker decides to do. In any case, they probably need to be rebuilt with VC14 unless the authors have very carefully kept a clean API, in which case it may as well be a DLL.

Because we're talking about a plugin architecture here, I think it's actually advantageous to use static linking of the C++ runtime. The main difference from the C runtime is that the C++ runtime does not have to be shared with the host - Python - and the advantage is that state will not be shared with other plugins that also happen to use the same version of C++ (or worse, a different version with the same name, and now we have a conflict).

I appreciate the problems this causes when trying to link in 3rd-party dependencies, but given a choice between making life easier for developers or users I have to side with the users (while doing as much as I possibly can to make developers lives easier). People installing wheels from Christoph's page or PyPI should be able to expect it to work. When pip grows extensions I'll certainly look into writing an extension for specifying, detecting and getting the required VC redistributable, but I don't want core Python to be burdened with shipping the full set of distributables.

[malemburg]

On 17.08.2015 18:24, Steve Dower wrote:

We can't have Python run VCredist because it requires administrative privileges and installing Python does not require those. This is actually one of the biggest complaints about the current installer, but because it looks like the design and not a bug it only comes up when you ask people directly what they'd like to change. (The most common bug is pip failing to install/uninstall, which is what you see on the issue tracker.)

Why not ? People certainly trust the Python installer more than some script which might want to do this afterwards in order to get a package working. After all, it's signed by the PSF and comes with certificate backed verification of authenticity.

The other benefits are that PYDs compiled with VS 2016+ (or whatever it's called) with these options will work with stock Python 3.5 on a clean machine - so if your extension links to python3.dll you can build once for all Python versions 3.5 and later and know that it will just copy and run. If you're building from source and only have a newer compiler than VC14, you will be able to build successfully. Also, we could decide to build some future Python 3.5.x with a newer compiler as it should not break any existing extensions (after much discussion and testing, obviously, but it is only feasible if we properly hide the VC version now).

This would only work for extensions using the stable Python ABI. The standard approach is to rebuild extension for every minor release, since the standard Python ABI is not guaranteed to stay backwards compatible.

To achieve these (apart from the last point) with VCredist, we need to be installing the current and all future versions of the redist with Python. Hence a time machine is required, and I haven't come up with any good way to simulate a time machine here.

See above. This is really not a very common use case. If you search for Py_LIMITED_API (which has to be defined to enable the stable ABI), you'll hardly find any reference to code using it.

> not when even MS itself recommends against doing this.

I have separate advice (also from MS, from the same people who have been quoted previously, but in private conversations so I can't link to it) that if we want any chance of our plugin architecture being VC version independent, this is what we have to do. I'm emailing again to get more specific advice, but the official guidance has always been biased by wanting people to get the latest tools (which is why VC9 disappeared until my team made the case that sometimes people can't upgrade). We're still pushing hard to make this an acknowledged use case, and don't be surprised if at some point in the future official advice says "if you allow plugins, do what CPython did to help your users and developers".

Regardless of marketing strategies, the fact that you have to reinstall Python and all extensions in case there's some bug in the CRT is really the main argument against doing static linking.

Static linking of the CRT is normally only used in situations where you don't want to have single file executables without external dependencies, e.g. for working on arbitrary Windows systems without having to install anything. It's a valid use case, but not a general purpose one.

The "/MT" == "statically linked" equivalence is an oversimplification in the presence of link-time code generation ("/GL" and "/LTCG" options), as we can take .obj or .lib files compiled with /MT and still use dynamic linking. The difference is we have to do it explicitly, which is what the "/nodefaultlib:libucrt.lib ucrt.lib" options do. If we add concrt140, msvcp140 and vccorlib140 to that as well (and probably the rest of the versions redistributables) then all of them will continue to be dynamically linked.

So basically, all the existing static libs could be built with /MT and still have their dependencies dynamically linked if that's what the final linker decides to do. In any case, they probably need to be rebuilt with VC14 unless the authors have very carefully kept a clean API, in which case it may as well be a DLL.

You lost me there. What's the advantage of using /MT when you then add all of the CRT libs to the set of libs which are dynamically linked ?

Just to clarify:

If I want to ship a C extension compiled for Python 3.5 which links to an external DLL on the system, I will have to tell the users of the extension to first run VCredist in order for them to be able to use extension on their system, since Python 3.5 will not ship with the necessary CRT DLLs, correct ?

Because we're talking about a plugin architecture here, I think it's actually advantageous to use static linking of the C++ runtime. The main difference from the C runtime is that the C++ runtime does not have to be shared with the host - Python - and the advantage is that state will not be shared with other plugins that also happen to use the same version of C++ (or worse, a different version with the same name, and now we have a conflict).

Python C extensions are much more than a plugin architecture. They allow connecting Python to the world of available C libraries out there, not just ones which you can statically compile.

I appreciate the problems this causes when trying to link in 3rd-party dependencies, but given a choice between making life easier for developers or users I have to side with the users (while doing as much as I possibly can to make developers lives easier). People installing wheels from Christoph's page or PyPI should be able to expect it to work. When pip grows extensions I'll certainly look into writing an extension for specifying, detecting and getting the required VC redistributable, but I don't want core Python to be burdened with shipping the full set of distributables.

Same here, and even more so: I would like the users to get a Python installation they can use out of the box, without having to worry about DLL hell and this includes making it possible to install Python C extensions with external DLL dependencies without the need to run VCredist before being able to do so and it includes getting CRT bug fixes by means of OS updates rather than complete reinstalls of Python and all your extensions.

Developers can work around these things, but if we end up requiring them to redistribute VCredist with every single package that has external dependencies, just to be able to install a binary package, because Python itself doesn't ship the necessary DLLs, then something is clearly broken for Python on Windows.

Users will also likely not grant pip admin privileges to run VCredist for them (or ask their sys admins to allow pip to do this), so that idea is not going to work out in practice.

Alternatively, we tell users to install VCredist by hand in case they plan to use Python with C extensions. Possible, but not very user friendly either.

With Python 2.7 all this is not an issue, creating yet another road block to prevent upgrades :-(

I understand that you would like us to get rid off the compiler version dependency and appreciate your work in that direction, but chances are high that this will not actually work out in practice. Backwards compatibility is already hard, forward compatibility is really really difficult to achieve without a time machine.

BTW: Do you know whether Python 3.5 will still work as Windows service without the MSVCR140 DLL available in the system folder ?

[zooba]

Why not [require administrative privileges]?

Because some people want to use Python and don't have administrative permissions on their own machine (e.g. students, employees, researchers, etc.)

The standard approach is to rebuild extension for every minor release, since the standard Python ABI is not guaranteed to stay backwards compatible.

The last point I posted lets you rebuild extensions *with a different compiler*. So when you upgrade to VC15 you can use that to build your Python 3.5 extensions instead of VC14, and still ship those extensions to users without requiring them to install the VC15 redist. Static linking is part of this

the fact that you have to reinstall Python and all extensions in case there's some bug in the CRT is really the main argument against doing static linking.

The same argument applies against app-local copies of the runtime, which is why I'm trying to avoid these dependencies rather than volunteering to redistribute and update them for all extensions.

Also, we already rebuild and release Python due to bugs in OpenSSL, so I see no reason why we wouldn't do the same for bugs in the (small) part of the CRT we statically link.

Static linking of the CRT is normally only used in situations where you don't want to have single file executables without external dependencies, e.g. for working on arbitrary Windows systems without having to install anything. It's a valid use case, but not a general purpose one.

Normally, yes, but we have a new use case now that the majority of the C runtime is preinstalled on machines and versionless.

What's the advantage of using /MT when you then add all of the CRT libs to the set of libs which are dynamically linked ?

The part that is statically linked is only a few functions that are either critical to program initialization or specially optimized by the compiler, and the rest of the CRT is versionless and managed by the operating system.

To be clear, the DLLs I listed are not required by CPython or most pure-C extensions (including, I assume, cffi and Cython). It's only when you start using C++ and especially the Microsoft provided libraries for parallelism that you need these DLLs.

If I want to ship a C extension compiled for Python 3.5 which links to an external DLL on the system, I will have to tell the users of the extension to first run VCredist in order for them to be able to use extension on their system, since Python 3.5 will not ship with the necessary CRT DLLs, correct ?

Correct, just as it's always been. However, if you want to ship a C extension compiled for Python 3.5 which *does not link to an external DLL, you *do not have to tell your user to install anything.

includes getting CRT bug fixes by means of OS updates rather than complete reinstalls of Python and all your extensions.

Users of Python 3.5 will get CRT updates from the OS. Only a few trivial functions are statically linked under this scheme.

Developers can work around these things, but if we end up requiring them to redistribute VCredist with every single package that has external dependencies, just to be able to install a binary package, because Python itself doesn't ship the necessary DLLs, then something is clearly broken for Python on Windows.

Alternatively, we tell users to install VCredist by hand in case they plan to use Python with C extensions. Possible, but not very user friendly either.

I proposed above making this an option in the installer. Problem is, it doesn't help extensions built with later compilers. Only the extension knows which version of the redist is needed.

With Python 2.7 all this is not an issue, creating yet another road block to prevent upgrades :-(

The issue with Python 2.7 is that you need to use VC9. If you think requiring everyone to use VC14 for Python 3.5 forever is less of an issue than having to recompile static libraries with a new version of the compiler, please say so explicitly, as I have not heard anyone claim that and need to hear it said in complete seriousness before I have any chance of believing it's true.

chances are high that this will not actually work out in practice. Backwards compatibility is already hard, forward compatibility is really really difficult to achieve without a time machine.

Agreed. However, the forward compatibility problem we're facing right now seems to be external libraries compiled under old compilers - that is, VC10 (to pick one example) failed at forward compatibility, and some work is required to fix that failure. If we concede defeat now, then we will also fail at forward compatibility, and it will take work in the future to correct our failure. I'm trying to do as much work now as I can to offset that work for everyone else later (and thanks for holding me accountable on this - it's more helpful than my apparently frustrated responses probably make it seem :) ).

BTW: Do you know whether Python 3.5 will still work as Windows service without the MSVCR140 DLL available in the system folder ?

Yes, it will. MSVCR140 does not exist - it is called ucrtbase.dll and ships with Python 3.5, Windows 10, and (later this year, I'm told) through normal Windows updates for downlevel operating systems. This is the *only* part of the CRT (it's also the majority of the CRT) that we install, and we do force users to have administrative privileges when installing Python if they don't already have it.

Python does not depend on the C++ runtime. If an extension requires the C++ runtime, users will have to acquire it some other way.

[50363613-ac32-4562-87b1-ef078e34ece0]

Two findings regarding the new "semi-static linking" options:

Distutils now creates libraries (.lib) that "may not be readable by subsequent versions of Visual C++" \https://msdn.microsoft.com/en-us/library/0zza0de8.aspx\.

Build times and static library sizes significantly increase to a point where it becomes impractical in some cases. For example, on my system, the zeromq package builds in 90 s on Python 3.4.3, while on Python 3.5rc1 it takes 470 s. Building HDF5 static libs with the new options takes 30 minutes (close to 100% CPU usage on 8 logical processors) compared to less than 5 minutes for the default options.

[zooba]

Do you know where that time is being spent?

I'd guess it's in link.exe (or lib.exe) while it displays a "generating code" message. You should be able to omit the "/LTCG" option from lib.exe when building a static library, though I suspect that will push the delay into the final link step when building the DLL/PYD.

And while it takes longer, it's almost certainly doing a better job of optimizing everything (I'd guess it's probably O(N**2) with the number of obj files).

[50363613-ac32-4562-87b1-ef078e34ece0]

Do you know where that time is being spent?

The incremental linker.

I'd guess it's probably O(N**2) with the number of obj file

Doesn't seem so. For example the pyrxp 2.1 package contains only 23 C files and takes minutes to link. Most packages compile and link reasonably fast.

[50363613-ac32-4562-87b1-ef078e34ece0]

I have now recompiled hundreds of Python packages and C/C++/Fortran libraries with the /MT /GL /LTCG /NODEFAULTLIB:libucrt.lib ucrt.lib flags. Many packages test OK, only few crashes.

Some more findings:

The /MT flag forces to also statically link the Intel Fortran runtime library.

Some libraries, e.g. OpenCV and Boost DLLs, can not be built with those flags (must be /MD).

Although most packages pass unittests (numpy, scipy, matplotlib, etc.), running real world scripts, Pandas unit tests, or simple scripts in IPython notebook, fail semi-randomly with "ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed". This is reproducible with Python 3.5.0rc2, 64 and 32 bit, on Windows 10 and 7, 64 bit. It seems to be related to the number of extensions and DLLs loaded into the process. This needs more testing. Is there still a limit of how many DLLs, which are statically-linked to the CRT, can be loaded? \https://support.microsoft.com/en-us/kb/193462\ \http://stackoverflow.com/questions/1437422\ \https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/3546c3c4-1b36-4552-85c5-1b3ba860ee84\

[zooba]

It may be possible to dynamically link the Fortran library in a similar fashion, though it depends on what code the compiler generates. For Python's purposes, statically linking it probably isn't a bad idea.

I'll have to check out whether the FLS limitation still applies, but it seems likely. I wasn't aware of it.

[zooba]

The FLS limit is definitely still there, and it still seems to be 127, which would explain the issues.

Best fix I have for this is to build with /MD and force vcruntimeXXX.dll to always be copied alongside builds. That will solve the installation issue and the DLL limit, at the cost of complicating licensing and bundling for packages. (The DLL *should* only be loaded once, even if it appears in multiple locations.)

I don't think this is necessarily a release blocker though. It may just be better for specific packages to do it - e.g. numpy could include whichever vcruntime DLL was used to at least satisfy all of its PYDs on any Python version. If someone happens to have matched builds of numpy/scipy/pandas (which seems fairly likely) and they're all /MD and bundle vcruntime then that's one FLS index for a decent number of DLLs.

A "better" solution might involve a pip extension that can detect which VC redist version is required when extracting a wheel and download/install it for the user (I considered proposing this closer to Python's core, but it really needs to be at download/install time and not load time).

Package distros and "packs" are likely to be in a position to distribute the dependencies they need and can avoid it. Unfortunately, I don't see a way to do that from core without effectively restricting the VC version for most people. Hopefully one day I'll get to deal with some easy problems... :)

[50363613-ac32-4562-87b1-ef078e34ece0]

Just to confirm: the following script fails with ImportError: DLL load failed on Python 3.5.0rc2. It fails around the 120th extension. The script passes on Python 3.4.3.

Also, Python 3.5.0rc2 (not Python 3.4.3) requires the extra_postargs=['/DLL'] argument to the link_shared_lib function, otherwise the linker fails with LINK : fatal error LNK1561: entry point must be defined.

# test_issue24872.py
#
# Build up to 256 C extension modules and import them.
#
# On Python 3.5.0rc2 for Windows this raises "ImportError: DLL load failed:
#   A dynamic link library (DLL) initialization routine failed."
#
# http://bugs.python.org/issue24872

import os
import sys
from importlib import import_module
from distutils import ccompiler

assert sys.platform == 'win32'
assert sys.version_info > (3, 2)

c_source = """/* Minimal Python 3 extension module */
#include "Python.h"
static struct PyModuleDef moduledef = {
    PyModuleDef_HEAD_INIT, "%s", NULL, -1, NULL, NULL, NULL, NULL, NULL
};
PyMODINIT_FUNC PyInit_%s(void) { return PyModule_Create(&moduledef); }
"""

cl = ccompiler.new_compiler()

for i in range(256):
    name = '_tmp%.3i' % i
    try:
        os.remove(name+'.pyd')
    except FileNotFoundError:
        pass
    with open(name+'.c', 'w') as fh:
        fh.write(c_source % (name, name))
    objects = cl.compile([name+'.c'])
    cl.link_shared_lib(objects, output_libname=name, extra_postargs=['/DLL'],
                       export_symbols=["PyInit_"+name])
    for ext in 'c', 'obj', 'exp', 'lib':
        os.remove(name+'.'+ext)
    os.rename(name+'.dll', name+'.pyd')
    import_module(name)

[340ae65e-9726-4646-a1b5-7bce30dece82]

still have the link.exe 1561 error without extra_args=['/DLL'] issue. is there a patch? It goes in distutils._msvccompiler right?