search

AlabamaASRL / asset_asrl

https://alabamaasrl.github.io/asset_asrl/
Apache License 2.0
30 stars 6 forks source link

Silent Death for Windows Binaries Built with Latest Version of Visual Studio #62

Closed jbpezent closed 3 months ago

jbpezent commented 3 months ago

I'm seeing silent death for certain scripts running windows binaries recently uploaded to PyPi (version 0.4.0 and 0.5.0). Binaries compiled from source locally on windows have no issues and neither do any linux binaries hosted on PyPi.

jbpezent commented 3 months ago

Unfortunately, I've narrowed it down to a recent update to the MSVC build tools that ship with visual studio and are used in the GitHub actions runners. I was able to reproduce the bad binaries when building locally with the latest version of visual studio (v17.10.3). I don't currently think that it is anything to do with our code base, but I will investigate. In the mean-time people building from source on windows should not update their MSVC build tools or update visual studio if everything is working right now.

jbpezent commented 3 months ago

Some progress on workaround. Downgrading the MSVC++ platform toolset from 14.40 to 14.29 in Github Actions results in working binaries. Also looks like this toolset can be installed side by side 14.40 locally in visual studio by installing the extension below.

image

Visual studio wont use it by default, so its not a fix for local builds yet. I'll figure out how to set that up and add it to the build tutorial.

jbpezent commented 3 months ago

Offending windows wheels for version 0.4 and 0.5 have been removed from PyPi. Linux ones are still available. We will do a new windows release soon.

jbpezent commented 3 months ago

No need to downgrade tools. Looks like this issue is caused by a specific update to std::mutex in the latest version of msvcp140.dll. See link for details.

The bad builds would work fine if they were running with the latest msvcp140.dll. However anaconda and other python distros ship with older incompatible versions, hence the crashes. I can confirm that compiling our binaries with _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR as suggested by msft fixes the issue even when running against old dlls. That is what we will do for now.