numba v0.38.0

[regro-cf-autotick-bot]

Fixes https://github.com/conda-forge/numba-feedstock/issues/4

This PR was created by the cf-regro-autotick-bot.

The cf-regro-autotick-bot is a service to automatically track the dependency graph and propose package version updates for conda-forge. It is very likely that the current package version for this feedstock is out of date.

Notes and instructions for merging this PR:

1. Please check that the dependencies have not changed.
2. Please merge the PR only after the tests have passed.
3. Feel free to push to the bot's branch to update this PR if needed.
4. The bot will almost always only open one PR per version.

If you would like a local version of this bot, you might consider using rever. Rever is a tool for automating software releases and forms the backbone of the bot's conda-forge PRing capability. Rever is both conda (conda install -c conda-forge rever) and pip (pip install re-ver) installable.

Finally, feel free to drop us a line if there are any issues!

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[CJ-Wright]

needs llvm update.

[jakirkham]

It needs PR ( https://github.com/conda-forge/llvmlite-feedstock/pull/2 ) specifically. Currently working through the backlog of PRs/issues here. So will try to deal with those first before going to this version.

[jakirkham]

@conda-forge-admin, please re-render.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-webservice.

I rerendered the feedstock and it seems to be already up-to-date.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jakirkham]

@conda-forge-admin, please re-render.

[jakirkham]

Windows 64-bit Python 2.7 fails as there appears to be no builds of llvmlite 0.23.0 (and we don't build those in conda-forge currently). Raised issue ( https://github.com/numba/llvmlite/issues/352 ) upstream to get a timeline as to when those builds might be made available.

[jakirkham]

Now that PR ( https://github.com/conda-forge/llvmlite-feedstock/pull/3 ) has been merged and built new llvmlite packages based off of llvmdev built with PR ( https://github.com/conda-forge/llvmdev-feedstock/pull/35 ), we are ready to restart CIs here.

[jakirkham]

Disabled Windows again and merged as macOS builds and all Linux Python 3 builds passed on commit ( https://github.com/conda-forge/numba-feedstock/pull/7/commits/f3b6189deaa30f572437de7269e7b6d007ec0dfd ). The patch only served to fix Python 2.7, which it appears to have done based on testing of commit ( https://github.com/conda-forge/numba-feedstock/pull/7/commits/b8e106e81de73e980378846f68f20058a125d2f7 ). The latest commits merely disabled Windows, which doesn't need to be tested more thoroughly.

[stuartarchibald]

Assuming this is a suitable log for diagnosing errors on windows: https://ci.appveyor.com/project/conda-forge/numba-feedstock/build/1.0.10/job/tbt52ea0yarvyjk5 (from https://github.com/numba/numba/issues/2925). This looks like Windows 64bit with Python 3.6? On the Numba build farm this passes all tests. Looking at the fails, below are guesses to the causes of the failures/suggestions/comments:

• [01:07:21] FAIL: test_linalg_cond (numba.tests.test_linalg.TestLinalgCond)
  [01:07:21] FAIL: test_linalg_det (numba.tests.test_linalg.TestLinalgDetAndSlogdet)
  [01:07:21] FAIL: test_linalg_slogdet (numba.tests.test_linalg.TestLinalgDetAndSlogdet)
  [01:07:21] FAIL: test_linalg_matrix_power (numba.tests.test_linalg.TestLinalgMatrixPower)

  The above tests all use LAPACK LU decomposition ({s,d,c,z}getrf) at some point and are operating on complex data types. If some instructions on how to reproduce on a local machine are available I can take a look.

• [01:07:21] FAIL: test_svml (numba.tests.test_svml.TestSVML)

  This is because SVML is not working but was detected as being present. I think this is due to llvmdev not being patched to use SVML (patch is here), but on windows icc_rt is needed (which contains svml_dispmd) as it provides components for the compilation chain for NumPy. Hence, the SVML library is detected and loaded, llvm is instructed to use it, but the generated code doesn't contain reference to the intrinsics as llvm is not patched to generate them.

• [01:07:21] FAIL: test_expm1_D_D (numba.tests.test_ufuncs.TestLoopTypesComplexNoPython)
  [01:07:21] FAIL: test_expm1_F_F (numba.tests.test_ufuncs.TestLoopTypesComplexNoPython)

  Fails for complex data types and differences are large, this is suspiciously similar to a dispatch issue I recently saw on ARMv7 (as are the linalg fails above come to think of it).

It may also be worth:

• Adding numba -s to the meta.yaml test::commands section. This diagnostic tool reports a lot of information about the machine set up and environment in a concise manner that is useful for developers when debugging. Example output is here: https://ci.appveyor.com/project/pitrou/numba/build/1.0.3146#L252
• Calling the test suite as per the Numba recipes, e.g. https://github.com/numba/numba/blob/master/buildscripts/condarecipe.local/run_test.sh#L15-L19. This first tests that the test discovery is working and that the test suite is capable of running, then it runs the test suite but does so using multiple processes such that it completes more quickly.
• Adding in the optional run time dependencies to the testing section like https://github.com/numba/numba/blob/master/buildscripts/condarecipe.local/meta.yaml#L38-L47 such that the whole test suite is exercised.

[jakirkham]

Thanks @stuartarchibald. This is very helpful. :)

Ah ok, was not sure if Numba was dispatching to BLAS/LAPACK or if it was compiling these codes itself. So right now we are using numpy built with openblas on Windows, which used Flang to build the Fortran parts. While we do run the full numpy test suite, a build issue with numpy or openblas is not totally out of the question. Guess the first step would be to isolate whether NumPy itself is behaving correctly. If NumPy is working ok, feel free to dive in. Running conda create -n test -c conda-forge python=3.6 blas numpy numba would be a good starting point. Feel free to add other tools in as needed.

Was aware of the Intel SVML patch, but haven't patched llvmdev with it yet. Opened issue ( https://github.com/conda-forge/llvmdev-feedstock/issues/36 ) to see if that will be agreeable. If you could share a bit about the patch in that issue, I think that would help.

Thanks for the debugging tips. Will give them a try.

Is there a way to restrict how many processes are used for testing? Sometimes CIs are not reliable at providing this information themselves. So we often have to override this manually.

Do you have a rough idea of how long it takes for the optional tests to run?

[stuartarchibald]

No problem! Glad this helps.

Numba grabs the pointers from the cython exported PyCapsules of LAPACK/BLAS functions coming from SciPy, so whatever SciPy is using for its LAPACK/BLAS Numba will too. Typically NumPy and SciPy both have to be behaving correctly for Numba to be totally happy. I'll take a look at this when I get a chance :)

IIRC parallel testing is just using a multiprocessing.Pool(), I don't see a reason why it couldn't take a CPU count as an arg though, PRs are welcome :)

I'm not sure how long the optional tests take to run I'm afraid, whilst they are "optional" cffi/ctypes and np.linalg support are fairly critical so we pretty much always run them.

[jakirkham]

Numba grabs the pointers from the cython exported PyCapsules of LAPACK/BLAS functions coming from SciPy, so whatever SciPy is using for its LAPACK/BLAS Numba will too. Typically NumPy and SciPy both have to be behaving correctly for Numba to be totally happy. I'll take a look at this when I get a chance :)

This is very important info. Thanks for this.

Have had similar experiences with NumPy and SciPy in the past. In particular, needing them to have the same LAPACK/BLAS combination (especially for software that relied on both like scikit-learn). Didn't realize that Numba was making use of SciPy's Cython bindings to LAPACK/BLAS.

Given this info, suspect NumPy/SciPy issues may be this is part of the problem. We should check before diving in too deep. In particular, NumPy is built with OpenBLAS. However people ran into problems trying to build SciPy with OpenBLAS. Work occurred along these lines in PR ( https://github.com/conda-forge/scipy-feedstock/pull/78 ). So SciPy is still using MKL currently.

While this generally has not caused problems for other software that uses both, Numba is a bit of a power user in this regard. So wouldn't be surprised if the problems we are seeing are a consequence of the different LAPACK/BLAS choice in NumPy and SciPy. Happy to be proven wrong here though.

IIRC parallel testing is just using a multiprocessing.Pool(), I don't see a reason why it couldn't take a CPU count as an arg though, PRs are welcome :)

That's good to hear. Probably won't have time to do anything about it near term, but have noted this need with issue ( https://github.com/numba/numba/issues/2940 ) for now.

I'm not sure how long the optional tests take to run I'm afraid, whilst they are "optional" cffi/ctypes and np.linalg support are fairly critical so we pretty much always run them.

Ok, will look into it.

Thanks again for your help.