Failed to import onnx package built with runner version: '2.317.0' (Version: 20240729.2.0)

[liqunfu]

Description

Failed to import onnx package built with runner version: '2.317.0' (Version: 20240729.2.0): ImportError: DLL load failed while importing onnx_cpp2py_export: A dynamic link library (DLL) initialization routine failed.

https://github.com/onnx/onnx/issues/6267 https://github.com/onnx/onnx/actions/runs/10240391088/job/28327161201

It could be any package built with the runner version.

Platforms affected

• [ ] Azure DevOps
• [X] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] Ubuntu 24.04
• [ ] macOS 12
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [ ] macOS 14
• [ ] macOS 14 Arm64
• [ ] Windows Server 2019
• [X] Windows Server 2022

Image version and build link

pip install onnx import onnx

Is it regression?

runner version: 'Version: 20240721.1.0': https://github.com/onnx/onnx/actions/runs/10203677822/job/28230518589

Expected behavior

shall be able to import the onnx package

Actual behavior

import onnx failed with: ImportError: DLL load failed while importing onnx_cpp2py_export: A dynamic link library (DLL) initialization routine failed.

Repro steps

Step1. pip install onnx Step2. import onnx

[Prabhatkumar59]

Hi @liqunfu Thank you for bringing this issue to our attention. We will look into this issue and will update you on this issue after investigating

[kishorekumar-anchala]

Hi @liqunfu ,

Could you please try with new image rollout (20240804.1.0) , it is working . Thank you !

[liqunfu]

Hi @liqunfu ,

Could you please try with new image rollout (20240804.1.0) , it is working . Thank you !

I ran the CI - it is still using: Image: windows-2022 Version: 20240729.2.0 and is still failing (https://github.com/onnx/onnx/actions/runs/10305222956/job/28525511662). Looking to released runner images (https://github.com/actions/runner-images/releases?page=1) I do not see 20240804.1.0 for Windows 2022 server. BTW, if the new release if available the CI will pick it up automatically, right?

[kishorekumar-anchala]

Hi @liqunfu , Could you please try with new image rollout (20240804.1.0) , it is working . Thank you !

I ran the CI - it is still using: Image: windows-2022 Version: 20240729.2.0 and is still failing (https://github.com/onnx/onnx/actions/runs/10305222956/job/28525511662). Looking to released runner images (https://github.com/actions/runner-images/releases?page=1) I do not see 20240804.1.0 for Windows 2022 server. BTW, if the new release if available the CI will pick it up automatically, right?

@liqunfu ,

Sorry, my bad. i thought it is for ubuntu 22 , will look into it and confirm you.

[kishorekumar-anchala]

@liqunfu ,

Could you please try with new image rollout (20240807.2) .Thank you !

[liqunfu]

@liqunfu ,

Could you please try with new image rollout (20240807.2) .Thank you !

It is still not working: https://github.com/onnx/onnx/actions/runs/10372781946/job/28716315392. How can we choose an older runner image to build the release so that we can publish a patch before a new runner with fix become available? BTW, it must be related to the c++ binary - we need to do this to work around the problem: "conda install conda-forge::vs2015_runtime". workaround with vc2015 install

[kishorekumar-anchala]

HI @liqunfu ,

vs2015_runtime will may not support with windows-latest image , have you tried with windows-2019 image?

[liqunfu]

HI @liqunfu ,

vs2015_runtime will may not support with windows-latest image , have you tried with windows-2019 image?

vs2015_runtime dependency was introduced with the new runner image. runner image before 20240721.1.0 produced ONNX package without vs2015_runtime dependency therefore it could be imported without any issue.

[liqunfu]

Did this happen again with newer runner image: "they updated the compiler without updating the VCRedist package."? https://github.com/actions/runner-images/issues/10020#issuecomment-2156793483

[liqunfu]

@kishorekumar-anchala We are close to release ONNX 1.17.0 and is block by this issue. Is there a way to specify a older version of Action runner image. The latest Windows 2022 action runner has a runtime newer than most of python users' thus the package built with the latest runner is not compatible with most python environments used by our users. In general, in order for a python package to be compatible with users' environment, it is better to build the package with an older runtime. GitHub action only allows to use the latest runner which is a problem for any repo that is building python packages.

[kishorekumar-anchala]

@kishorekumar-anchala We are close to release ONNX 1.17.0 and is block by this issue. Is there a way to specify a older version of Action runner image. The latest Windows 2022 action runner has a runtime newer than most of python users' thus the package built with the latest runner is not compatible with most python environments used by our users. In general, in order for a python package to be compatible with users' environment, it is better to build the package with an older runtime. GitHub action only allows to use the latest runner which is a problem for any repo that is building python packages.

1) Older version of runner image is not possible. 2) Could you please share python runtime or version to check.

[liqunfu]

Here is the environment with which I can reproduce the issue: vc 14.2 h21ff451_1 vs2015_runtime 14.27.29016 h5e58377_2

The runner (https://github.com/onnx/onnx/actions/runs/10460371160/job/28966396210) has VC version: C++ compiler : C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe -- C++ compiler version : 19.40.33813.0

[liqunfu]

This issue is due to a breaking change in https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-notes-v17.10 (which maps to compiler version 14.40). Users need to upgrade msvc runtime to the latest to fix this dll loading failure. I have added fix/workaround to ONNX documentation. Close this issue.