search

conda-forge / cupy-feedstock

A conda-smithy repository for cupy.
BSD 3-Clause "New" or "Revised" License
5 stars 23 forks source link

cupy v8.6.0 #113

Closed regro-cf-autotick-bot closed 3 years ago

regro-cf-autotick-bot commented 3 years ago

Close #62. Close #109.

It is very likely that the current package version for this feedstock is out of date. Notes for merging this PR:

  1. Feel free to push to the bot's branch to update this PR if needed.
  2. The bot will almost always only open one PR per version. Checklist before merging this PR:
    • [ ] Dependencies have been updated if changed: see upstream
    • [ ] Tests have passed
    • [ ] Updated license if changed and license_file is packaged

Note that the bot will stop issuing PRs if more than 3 Version bump PRs generated by the bot are open. If you don't want to package a particular version please close the PR.

NEW: If you want these PRs to be merged automatically, make an issue with code>@conda-forge-admin,</codeplease add bot automerge in the title and merge the resulting PR. This command will add our new bot automerge feature to your feedstock!

If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase code>@<space/conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. If you would like a local version of this bot, you might consider using rever. Rever is a tool for automating software releases and forms the backbone of the bot's conda-forge PRing capability. Rever is both conda (conda install -c conda-forge rever) and pip (pip install re-ver) installable. Finally, feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/autotick-bot/actions/runs/685248326, please use this URL for debugging

Dependency Analysis

We couldn't run dependency analysis due to an internal error in the bot. :( Help is very welcome!

regro-cf-autotick-bot commented 3 years ago

Hi! This feedstock was not able to be rerendered after the version update changes. I have pushed the version update changes anyways and am trying to rerender again with this comment. Hopefully you all can fix this!

@conda-forge-admin rerender

conda-forge-linter commented 3 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

github-actions[bot] commented 3 years ago

Hi! This is the friendly automated conda-forge-webservice. I tried to rerender for you but ran into some issues, please ping conda-forge/core for further assistance. You can also try re-rendering locally.

leofang commented 3 years ago

Need _wheels.json, but let me first see if c7961e8 can build.

leofang commented 3 years ago

btw because of an error I mentioned on Gitter, for Windows I cannot start with CUDA 10.0/10.1 (which zips with cuDNN 7), so I limit to CUDA 10.2+. Also, I took down the combination of cos6 + CUDA 10.1/10.2 for Linux. With this PR, the total number of variants become 58 (UPDATED) 54:

leofang commented 3 years ago

Noticed something odd: On Windows, the build would not terminate even if an error is generated (in this case I handled the env var NVCC incorrectly). Then, it proceeds to the test stage as usual, and import cupy fails...Something is wrong with the Windows pipeline.

leofang commented 3 years ago

@kmaehashi Windows builds successfully, but the test fails at the import stage: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=295513&view=logs&j=5be07ae1-d8ba-5406-47b6-8e3a3a12f825&t=0bf03e01-0bec-5b85-5316-b1633322e895&l=3903

Traceback (most recent call last):
  File "D:\bld\cupy_1616670932393\_test_env\lib\site-packages\cupy\__init__.py", line 20, in <module>
    from cupy import core  # NOQA
  File "D:\bld\cupy_1616670932393\_test_env\lib\site-packages\cupy\core\__init__.py", line 1, in <module>
    from cupy.core import core  # NOQA
  File "cupy\core\core.pyx", line 1, in init cupy.core.core
  File "D:\bld\cupy_1616670932393\_test_env\lib\site-packages\cupy\cuda\__init__.py", line 5, in <module>
    from cupy.cuda import compiler  # NOQA
  File "D:\bld\cupy_1616670932393\_test_env\lib\site-packages\cupy\cuda\compiler.py", line 12, in <module>
    from cupy.cuda import function
  File "cupy\cuda\function.pyx", line 1, in init cupy.cuda.function
  File "cupy\cuda\texture.pyx", line 1, in init cupy.cuda.texture
ImportError: DLL load failed: The specified module could not be found.

Any chance you've experienced this earlier?

leofang commented 3 years ago

It could also be related to how Conda handles dll on Windows...

leofang commented 3 years ago

@jakirkham Another thought: Unlike on Linux, we don't have a driver stub on Windows, so importing any modules linked to the driver will fail, right?

leofang commented 3 years ago

Note: the patch added in 05bcd2d was needed to resolve an error at this line: https://github.com/cupy/cupy/blob/30e004549b1862f3360b3b5922566bd6b9805bb2/cupy/_environment.py#L226 which I will backport to upstream.

leofang commented 3 years ago

Unlike on Linux, we don't have a driver stub on Windows, so importing any modules linked to the driver will fail, right?

I think this is what's happening. Will fix the test after the current run finishes.

leofang commented 3 years ago

I removed the combination of Windows + CUDA 11.2 due to a compiler error: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=295602&view=logs&j=84cf05f7-5267-5f38-137b-63208e75d05b&t=7c51bcc0-b6e0-5e4f-1968-7bb191e490bd&l=4336

    nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
    cupy_thrust.cu
    cupy/cuda/cupy_thrust.cu(145): error C3615: constexpr function 'thrust::less<thrust::complex<float>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(148): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(148): note: see usage of '_cmp_less'
    cupy/cuda/cupy_thrust.cu(157): error C3615: constexpr function 'thrust::less<thrust::complex<double>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(160): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(160): note: see usage of '_cmp_less'
    cupy/cuda/cupy_thrust.cu(169): error C3615: constexpr function 'thrust::less<thrust::tuple<unsigned __int64,thrust::complex<float>,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(172): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(172): note: see usage of '_tuple_less'
    cupy/cuda/cupy_thrust.cu(181): error C3615: constexpr function 'thrust::less<thrust::tuple<unsigned __int64,thrust::complex<double>,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(184): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(184): note: see usage of '_tuple_less'
    cupy/cuda/cupy_thrust.cu(223): error C3615: constexpr function 'thrust::less<float>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(226): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(226): note: see usage of '_real_less'
    cupy/cuda/cupy_thrust.cu(235): error C3615: constexpr function 'thrust::less<double>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(238): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(238): note: see usage of '_real_less'
    cupy/cuda/cupy_thrust.cu(247): error C3615: constexpr function 'thrust::less<thrust::tuple<unsigned __int64,float,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(250): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(250): note: see usage of '_tuple_less'
    cupy/cuda/cupy_thrust.cu(259): error C3615: constexpr function 'thrust::less<thrust::tuple<unsigned __int64,double,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(262): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(262): note: see usage of '_tuple_less'
    cupy/cuda/cupy_thrust.cu(287): error C3615: constexpr function 'thrust::less<__half>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(288): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(288): note: see usage of '_real_less'
    cupy/cuda/cupy_thrust.cu(297): error C3615: constexpr function 'thrust::less<thrust::tuple<unsigned __int64,__half,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type,thrust::null_type>>::operator ()' cannot result in a constant expression
    cupy/cuda/cupy_thrust.cu(300): note: failure was caused by call of undefined function or one not declared 'constexpr'
    cupy/cuda/cupy_thrust.cu(300): note: see usage of '_tuple_less'
    error: command 'D:\\bld\\cupy_1616682886118\\_build_env\\Library\\bin\\nvcc.bat' failed with exit status 2

I will create an issue upstream to fix this, and backport the fix here once it's done and verified.

jakirkham commented 3 years ago

@jakirkham Another thought: Unlike on Linux, we don't have a driver stub on Windows, so importing any modules linked to the driver will fail, right?

Sorry I have no clue. My knowledge of Windows is pretty poor unfortunately 😅

@jaimergp sorry to bug you, but do you know?

leofang commented 3 years ago

OK Windows tests are now passing in 5a4d36b. I propose we do this: Once all tests pass and everyone is happy, we release the packages, and someone can test the Windows package when they show up on Anaconda.org. If it works, great, but if not, we mark them as broken and then investigate.

@jakirkham please review the changes in all the dependency pinning stuff. @kmaehashi please review changes in the recipe folder, especially those enabling the preloading warning (_wheels.json).

leofang commented 3 years ago

Let me wait for all tests for the commit 5a4d36b to finish. After that, I want to test $CONDA_PREFIX\Library\bin for Windows, and if it doesn't help I'll revert back to 5a4d36b.

kmaehashi commented 3 years ago

Thank you for all the efforts to support Windows, @leofang!

Wheel metadatas itself LGTM, but are they also used for v8.6.0? I thought this dependency change happens in v9 (https://github.com/cupy/cupy/issues/4850) so packaging support is not backported to v8: https://github.com/cupy/cupy/pull/4873

Unfortunately not very confident about recipe/win64_detect_conda.diff as I don't have experience on conda + Windows. If you could share the built package locally I can setup conda on Windows and test it before it gets published to conda-forge repo.

jaimergp commented 3 years ago

I can help test on Windows! If you don't want to release directly, you can also enable the Azure artifacts and I will download the packages manually. Just let me know!

With respect to the Windows drivers, yes, there's no drivers or stubs. We investigated how to generate them procedurally on installation but since the drivers are not installed, we don't have access to the symbols and bringing those from "recent" versions require too much manual labor not to mention the potential EULA issues.

Hopefully some day we'll have an official Nvidia image on Azure, but so far this is what we get...

leofang commented 3 years ago

Wheel metadatas itself LGTM, but are they also used for v8.6.0? I thought this dependency change happens in v9 (cupy/cupy#4850) so packaging support is not backported to v8: cupy/cupy#4873

Well...ok I forgot 😅 So only v9 supports preloading of all libs, and v8 supports only preloading cuDNN? Anyway, what's currently done can be ported to v9.0rc1 (which I will try to work on shortly) and v9.0, so it's not a waste.

Unfortunately not very confident about recipe/win64_detect_conda.diff as I don't have experience on conda + Windows. If you could share the built package locally I can setup conda on Windows and test it before it gets published to conda-forge repo.

@kmaehashi If you have time to test it, it couldn't be better! I can turn on the artifact storage temporarily so that you can download them from Azure after the test finishes. How does that sound?

leofang commented 3 years ago

Thanks a lot @jaimergp!

I can help test on Windows! If you don't want to release directly, you can also enable the Azure artifacts and I will download the packages manually. Just let me know!

Great, will do that and ping you and Kenichi when it's ready.

With respect to the Windows drivers, yes, there's no drivers or stubs.

Thanks for confirming...So in any case import cupy on the Windows CI will just fail...

kmaehashi commented 3 years ago

Well...ok I forgot 😅 So only v9 supports preloading of all libs, and v8 supports only preloading cuDNN?

Yes, v8 preload supports cudnn (only), and it does not take care of conda things at all, so I think _wheel.json cannot be added for v8 conda-forge packages. https://github.com/cupy/cupy/blob/v8/cupy/_environment.py#L330

If you have time to test it, it couldn't be better! I can turn on the artifact storage temporarily so that you can download them from Azure after the test finishes. How does that sound?

Sounds good to me!

leofang commented 3 years ago

Given the above discussion, I think it's best if I clone this PR to target the rc branch (v9.0rc1), and then come back here to rework for v8.6, as the changes here were meant for v9.

leofang commented 3 years ago

5a4d36b is green

leofang commented 3 years ago

Wait, messed up CBC, will rebase fix and rerender.

leofang commented 3 years ago

All jsons are removed as preloading optional deps does not work on CuPy v8. As a result, I reverted to the old way of handling dependencies, which leads to 66 builds

This is ready for review.