Build Matplotlib 3.9.0rc1

dopplershift commented 4 months ago

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

conda-forge-webservices[bot] commented 4 months ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

tacaswell commented 4 months ago

@conda-forge-admin, please rerender

github-actions[bot] commented 4 months ago

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/matplotlib-feedstock/actions/runs/8842524900.

tacaswell commented 4 months ago

ImportError: /home/conda/feedstock_root/build_artifacts/matplotlib-suite_1714079322516/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/matplotlib/_path.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3agg10curve3_div4initEdddddd

That looks real 😱

tacaswell commented 4 months ago

It also looks like atleast some of the builds are trying to build our own freetype.

QuLogic commented 4 months ago

mplsetup.cfg is no longer used; config settings need to be passed in to the build backend. https://matplotlib.org/devdocs/api/next_api_changes/development/26621-ES.html

dopplershift commented 4 months ago

@qulogic Ah, I missed that.

QuLogic commented 3 months ago

Ping @dopplershift; I don't normally care about conda-forge packages, but some people have asked...

QuLogic commented 3 months ago

@conda-forge-admin please restart ci

conda-forge-webservices[bot] commented 3 months ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

This recipe is using a compiler, which now requires adding a build dependence on {{ stdlib("c") }} as well. Note that this rule applies to each output of the recipe using a compiler. For further details, please see https://github.com/conda-forge/conda-forge.github.io/issues/2102.
You're setting a constraint on the __osx virtual package directly; this should now be done by adding a build dependence on {{ stdlib("c") }}, and overriding c_stdlib_version in recipe/conda_build_config.yaml for the respective platform as necessary. For further details, please see https://github.com/conda-forge/conda-forge.github.io/issues/2102.
In your conda_build_config.yaml, please change the name of MACOSX_DEPLOYMENT_TARGET, to c_stdlib_version!

QuLogic commented 3 months ago

ImportError: /home/conda/feedstock_root/build_artifacts/matplotlib-suite_1714079322516/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/python3.10/site-packages/matplotlib/_path.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3agg10curve3_div4initEdddddd

That looks real 😱

We reproduced this at the the Summit the other day, and I've opened conda-forge/ctng-compilers-feedstock#135 to get that fixed in the compilers.

jakirkham commented 3 months ago

Based on discussion in the other thread, it looks like a configuration change is needed

Not entirely sure how the build choose the archiver, but one common way is checking what the environment variable AR is set to. So maybe something like this?

export AR="${BUILD_PREFIX}/bin/${HOST}-gcc-ar"

jakirkham commented 3 months ago

IIUC Meson is being used here. If so, it looks like it is possible to configure the ar used

QuLogic commented 2 months ago

It looks like GCC_AR is set to the right static linker, so we could set AR=$GCC_AR, but I'm hoping for the future Meson can just do that automatically https://github.com/mesonbuild/meson/issues/13324

tacaswell commented 2 months ago

@conda-forge-admin, please rerender

tacaswell commented 2 months ago

The windows builds are failing because something is looking for python 3.12 libraries to link to rather than the correct version.

It is also showing:

WARNING: Using legacy MSVC compiler setup. This will be removed in conda-build 4.0. If this recipe does not use a compiler, this message is safe to ignore. Otherwise, use {{compiler('')}} jinja2 in requirements/build.

but we have both

        - {{ compiler('c') }}
        - {{ compiler('cxx') }}

already.

tacaswell commented 2 months ago

  [91/101] Linking target src/_c_internal_utils.pypy39-pp73-win_amd64.pyd
  FAILED: src/_c_internal_utils.pypy39-pp73-win_amd64.pyd
  "link"  /MACHINE:x64 /OUT:src/_c_internal_utils.pypy39-pp73-win_amd64.pyd src/_c_internal_utils.pypy39-pp73-win_amd64.pyd.p/_c_internal_utils.cpp.obj "/release" "/nologo" "/OPT:REF" "/DLL" "/IMPLIB:src\_c_internal_utils.pypy39-pp73-win_amd64.lib" "ole32.lib" "shell32.lib" "user32.lib" "D:\bld\matplotlib-suite_1719418468749\_h_env\libs\python39.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"
  LINK : fatal error LNK1104: cannot open file 'python312.lib'

conda-forge-webservices[bot] commented 2 months ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

You're setting a constraint on the __osx virtual package directly; this should now be done by adding a build dependence on {{ stdlib("c") }}, and overriding c_stdlib_version in recipe/conda_build_config.yaml for the respective platform as necessary. For further details, please see https://github.com/conda-forge/conda-forge.github.io/issues/2102.
In your conda_build_config.yaml, please change the name of MACOSX_DEPLOYMENT_TARGET, to c_stdlib_version!

conda-forge-webservices[bot] commented 2 months ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

You are setting c_stdlib_version below the current global baseline in conda-forge (10.13). If this is your intention, you also need to override MACOSX_DEPLOYMENT_TARGET (with the same value) locally.

conda-forge-webservices[bot] commented 2 months ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

tacaswell commented 2 months ago

@conda-forge-admin, please rerender

tacaswell commented 2 months ago

well, I have increased the number of failures from when I started trying to fix this.

tacaswell commented 2 months ago

We are getting this on OSX now:

../meson.build:1:0: ERROR: Could not invoke sanity test executable: [Errno 86] Bad CPU type in executable: '/Users/runner/miniforge3/conda-bld/matplotlib-suite_1719431004502/work/.mesonpy-nxkm6vx1/meson-private/sanitycheckc.exe'.

which just leaves me confused.

jakirkham commented 2 months ago

Is there something we need to do for Windows builds to pick up freetype?

Seeing this on CI:

  Run-time dependency freetype2 found: NO (tried pkgconfig and cmake)

  ..\extern\meson.build:11:17: ERROR: Dependency "freetype2" not found, tried pkgconfig and cmake

jakirkham commented 2 months ago

@conda-forge-admin , please re-render

jakirkham commented 2 months ago

@conda-forge-admin , please re-render

jakirkham commented 2 months ago

@conda-forge-admin , please re-render

jakirkham commented 2 months ago

@conda-forge-admin , please re-render

jakirkham commented 2 months ago

@conda-forge-admin , please re-render

github-actions[bot] commented 2 months ago

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/matplotlib-feedstock/actions/runs/9688515882.

jakirkham commented 2 months ago

@conda-forge-admin , please re-render

jakirkham commented 2 months ago

Good news: All *nix builds (including cross-compiled ones) pass 🥳

Bad news: Windows is consistently failing. Having issues picking up system dependencies ( https://github.com/conda-forge/matplotlib-feedstock/pull/373#discussion_r1655870742 ) and before saw issues where builds tried to link the wrong Python library ( https://github.com/conda-forge/matplotlib-feedstock/pull/373#issuecomment-2192415806 ) (saw this with CPython as well) 😞

Not having access to a Windows machine, am ill-positioned to help with the latter. That said, hopefully it is now in a better place for someone to do that debugging 😬

jakirkham commented 2 months ago

Ok managed to resolve the system dependency issue on Windows

The linking issue remains. Here's an up-to-date example from CI

[30/40] Linking target src/_c_internal_utils.cp39-win_amd64.pyd
FAILED: src/_c_internal_utils.cp39-win_amd64.pyd 
"link"  /MACHINE:x64 /OUT:src/_c_internal_utils.cp39-win_amd64.pyd src/_c_internal_utils.cp39-win_amd64.pyd.p/_c_internal_utils.cpp.obj "/release" "/nologo" "/OPT:REF" "/DLL" "/IMPLIB:src\_c_internal_utils.cp39-win_amd64.lib" "ole32.lib" "shell32.lib" "user32.lib" "D:\bld\matplotlib-suite_1719474041159\_h_env\libs\python39.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"
LINK : fatal error LNK1104: cannot open file 'python312.lib'

Not sure why it is trying to link Python 3.12 libraries with Python 3.9 (or any other non-3.12 version)

QuLogic commented 2 months ago

The linker invocation doesn't even mention python312.lib:

"link"  /MACHINE:x64 /OUT:src/_c_internal_utils.cp39-win_amd64.pyd src/_c_internal_utils.cp39-win_amd64.pyd.p/_c_internal_utils.cpp.obj "/release" "/nologo" "/OPT:REF" "/DLL" "/IMPLIB:src\_c_internal_utils.cp39-win_amd64.lib" "ole32.lib" "shell32.lib" "user32.lib" "D:\bld\matplotlib-suite_1719474041159\_h_env\libs\python39.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"

I think this is coming in from one of the files; I have a vague memory that pybind11 adds the Python library via some kind of #pragma; do we need to use a Python-specific pybind11 somehow?

ianthomas23 commented 2 months ago

I think this is coming in from one of the files; I have a vague memory that pybind11 adds the Python library via some kind of #pragma; do we need to use a Python-specific pybind11 somehow?

This might not be relevant, but I have seen similar before on some build systems when meson, without specific guidance, picks the wrong build vs host version of pybind11. For possible workaround see https://github.com/conda-forge/contourpy-feedstock/blob/9f91c1552136d83d8e6e43c56e4c5f4ede64232f/recipe/build.sh#L12-L18 and https://github.com/emscripten-forge/recipes/blob/main/recipes/recipes_emscripten/contourpy/build.sh for example.

jakirkham commented 2 months ago

Yeah it is rather odd. Didn't see any other mentions in the logs either

Interesting observation about pybind11. Is this meaningful?

pybind11-config found: NO need ['>=2.6']
...
Run-time dependency pybind11 found: YES 2.13.1

Thought about cross-compilation. However as this is Windows and is being compiled natively, I don't think it is related. Though please let me know if I'm missing something

QuLogic commented 2 months ago

Or actually, this comes in from the Python headers: https://github.com/python/cpython/blob/1167a9a30b4b2f327ed987e845e378990d1ae6bf/PC/pyconfig.h.in#L310-L336 So we probably have some incorrect include path somewhere.

QuLogic commented 2 months ago

Maybe pass -Ccompile-args=-v to build, so that meson-python tells ninja to compile verbosely, and we can see what include path we have.

jakirkham commented 2 months ago

The path used for python39.lib here looks off. Would expect libs would Library\lib

Snippet from this CI run:

[30/40] "link"  /MACHINE:x64 /OUT:src/_c_internal_utils.cp39-win_amd64.pyd src/_c_internal_utils.cp39-win_amd64.pyd.p/_c_internal_utils.cpp.obj "/release" "/nologo" "/OPT:REF" "/DLL" "/IMPLIB:src\_c_internal_utils.cp39-win_amd64.lib" "ole32.lib" "shell32.lib" "user32.lib" "D:\bld\matplotlib-suite_1719486440797\_h_env\libs\python39.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"
FAILED: src/_c_internal_utils.cp39-win_amd64.pyd 
"link"  /MACHINE:x64 /OUT:src/_c_internal_utils.cp39-win_amd64.pyd src/_c_internal_utils.cp39-win_amd64.pyd.p/_c_internal_utils.cpp.obj "/release" "/nologo" "/OPT:REF" "/DLL" "/IMPLIB:src\_c_internal_utils.cp39-win_amd64.lib" "ole32.lib" "shell32.lib" "user32.lib" "D:\bld\matplotlib-suite_1719486440797\_h_env\libs\python39.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"
LINK : fatal error LNK1104: cannot open file 'python312.lib'

QuLogic commented 2 months ago

The corresponding compilation is:

[23/40] "cl.exe"
  "-Isrc\_c_internal_utils.pypy39-pp73-win_amd64.pyd.p"
  "-Isrc"
  "-I..\src"
  "-IC:/hostedtoolcache/windows/Python/3.12.4/x64/include"
  "-ID:/bld/matplotlib-suite_1719486398479/_h_env/Library/include"
  "-ID:\bld\matplotlib-suite_1719486398479\_h_env\Include"
  "-DNDEBUG"
  "/MD"
  "/nologo"
  "/showIncludes"
  "/utf-8"
  "/Zc:__cplusplus"
  "/W2"
  "/EHsc"
  "/std:c++17"
  "/permissive-"
  "/O2"
  "/Gw"
  "-DMS_WIN64="
  "/d2FH4-"
  "-DPY_ARRAY_UNIQUE_SYMBOL=MPL__c_internal_utils_ARRAY_API"
  "/Fdsrc\_c_internal_utils.pypy39-pp73-win_amd64.pyd.p\_c_internal_utils.cpp.pdb"
  /Fosrc/_c_internal_utils.pypy39-pp73-win_amd64.pyd.p/_c_internal_utils.cpp.obj
  "/c"
  ../src/_c_internal_utils.cpp

That early path of /IC:/hostedtoolcache/windows/Python/3.12.4/x64/include doesn't look right there.

Can we also check what's in %INCLUDE%?

jakirkham commented 2 months ago

From this build am seeing the following

INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.29.30133\include;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;D:\bld\matplotlib-suite_1719528423177\_h_env\Library\include
is_x64_arch=true
...
LIB=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64;D:\bld\matplotlib-suite_1719528423177\_h_env\Library\lib
LIBPATH=C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.29.30133\lib\x86\store\references;C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.22621.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.22621.0;C:\Windows\Microsoft.NET\Framework64\v4.0.30319

In all cases except LIBPATH it is adding the host environment path last

None of them reference hostedtoolcache. AFAICT it doesn't appear in any environment variable. Just in the flags. So unclear on how it got there

jakirkham commented 2 months ago

Ok all green 🥳

Please let me know if you need anything else here 🙂

jakirkham commented 2 months ago

Thanks Tom! 🙏

Looks like CI didn't start when this was merged. So have manually started it

That said, it looks like this may run into the same failure you observed in comment: https://github.com/conda-forge/matplotlib-feedstock/pull/390#issuecomment-2200503373

jakirkham commented 2 months ago

The macOS and Windows builds passed

Linux builds failed as expected

Since this issue emerged, we have deployed a fix. So restarting failed builds to see if the remaining issues clear up

jakirkham commented 2 months ago

All builds have now passed and uploaded packages

jtilly commented 2 months ago

Note that merging this PR into the rc branch of this repo pushed 3.9.0 builds onto conda-forge (see https://anaconda.org/conda-forge/matplotlib). Also see https://github.com/conda-forge/matplotlib-feedstock/pull/390#issuecomment-2208282126.

conda-forge / matplotlib-feedstock

Build Matplotlib 3.9.0rc1 #373