WHL: Window x64 builds failing

[snowman2]

https://github.com/pyproj4/pyproj/actions/runs/10307248459/job/28532125980

Error: Command pyproj -v && python -c "import pyproj; pyproj.Proj(init='epsg:4269')"  && cp -r D:\a\pyproj\pyproj/test . && python -m pytest test -v -s
 failed with code 3221225477. 

[jjimenezshaw]

That are 4 commands chained. We do not know which one is failing.

[EwoutH]

Seems like we need to untangle these commands:

https://github.com/pyproj4/pyproj/blob/2a27805c04f708328424f7e04809e96bb4d4e999/.github/workflows/release.yaml#L151-L155

Edit: Since it fails after over 30s, I'm quite sure it's the last line with pytest itself.

[EwoutH]

If you go to a failed run, like https://github.com/pyproj4/pyproj/actions/runs/10307248459/job/28532125980, then you should be able to rerun the failed job. You can checkmark the "run in debug" mode option, and then we might get some more information.

[jjimenezshaw]

No more info in debug :( https://github.com/pyproj4/pyproj/actions/runs/10307248459/job/28773570732

[EwoutH]

Can we run the tests separately, without the whole cibuildwheel thing? Or are those tests explicitly about the wheels?

If so, maybe remove the tests steps temporarily, download some of the wheels, and run them manually though pytest.

Or remove the test steps from cibuildwheel, and run them separately afterwards.

[snowman2]

Or are those tests explicitly about the wheels?

They are specifically for the wheels. They are meant to prevent publishing problematic wheels.

If so, maybe remove the tests steps temporarily, download some of the wheels, and run them manually though pytest.

I don't have a local Windows machine to test or debug with. The CI is how testing/debugging on Windows happens.

[snowman2]

If so, maybe remove the tests steps temporarily, download some of the wheels, and run them manually though pytest.

I believe can do that with a fork. The artifacts should be available in the completed action.

[djhoese]

FYI build is now called python-build in conda-forge so I had to update the release workflow. See:

https://github.com/djhoese/pyproj/commit/aa30fcd493f2cf9386995b4ec317a034d5902649

[djhoese]

Also, FYI the tests fail with just pyproj -v:

   + pyproj -v
  Error: Command pyproj -v failed with code 3221225477. 

[snowman2]

Potential backup if Windows wheels issue is not addressed: https://github.com/cgohlke/geospatial-wheels

[m-richards]

Just thought I'd share that I have a windows machine to test with and I've been having a bit of a look into this. If I skip this CI check on my fork and download the wheels from the gitlab CI artifact, I get a segfault at import time (modifying the wheel and adjusting the __init__.py with wheel unpack and wheel pack seems to suggest this happens for any of the calls into cython modules.

I can't replicate this locally though - building wheels myself using a similar setup vcpkg for proj deps, proj source code and msvc compiler produces a valid wheel which doesn't segfault. Happy to test other things if there are useful suggestions.

[rcomer]

Talk of segfaults reminds me of this recent Matplotlib issue https://github.com/matplotlib/matplotlib/issues/28551

I have no idea if the causes could be related though - I’m afraid that discussion went over my head.

[tacaswell]

I agree this sounds very similar.

The very short summary is that visual studio made a change to the c++ stdlibary. Things compiled before the change work with the new library, things compiled after change segfault when used with the old library. To try and be self-contained many wheels will bundle the c++ runtime. If you then have a wheel that bundles the old one, import it so that the old runtime gets loaded, and then use something compiled with the new version of visual studio you will get this segfault.

We also found that as part of the bundling process we would sometimes grab the wrong (and old) version of the dll.

[greglucas]

I was trying to add Python 3.13 wheels for Cartopy and are unable to do that because of the missing pyproj wheels for 3.13.

What was the recommended fix in that issue? Several different things were going on and I also did not follow the final recommendation on what wheel builders should be doing. Is it to statically link the msvc runtime? https://github.com/contourpy/contourpy/pull/427 or to do something with the PATH searching? https://github.com/matplotlib/matplotlib/issues/28551#issuecomment-2275391226 something else entirely?

[snowman2]

See #1439

[QuLogic]

What was the recommended fix in that issue? Several different things were going on and I also did not follow the final recommendation on what wheel builders should be doing. Is it to statically link the msvc runtime?

Yes, we statically linked the runtime: https://github.com/matplotlib/matplotlib/pull/28687