search

conda-forge / conda-forge.github.io

The conda-forge website.
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
131 stars 274 forks source link

NumPy 1.12 support for packages which require numpy x.x #313

Closed jjhelmus closed 6 years ago

jjhelmus commented 7 years ago

NumPy 1.12 was recently released. Packages that make use of the NumPy C API need to be built against a given minor NumPy version. In order for these packages to support NumPy 1.12, they must be rebuilt against NumPy 1.12.

A script (see next comment) is used to find packages that depend on numpy 1.11 but do not have a version which depends on numpy 1.12. This script is used to automatically generate the list below which will be updated as packages are rebuilt.

Packages that need to be updated to support NumPy 1.12:

jjhelmus commented 7 years ago

Script used to create checklist above:

#! /usr/bin/env python
from urllib.request import urlopen
import json

# fetch conda-forge repodata from URL
url = 'https://conda.anaconda.org/conda-forge/linux-64/repodata.json'
response = urlopen(url)
cf_index = json.loads(response.read().decode('utf8'))

# uncomment for a local repodata.json file
"""
with open('./repodata.json') as f:
    cf_index = json.load(f)
"""

# find packages that depend on numpy 1.11* and 1.12* (have numpy x.x)
pkgs = cf_index['packages']
np111 = set([p['name'] for p in pkgs.values() if 'numpy 1.11*' in p['depends']])
np112 = set([p['name'] for p in pkgs.values() if 'numpy 1.12*' in p['depends']])

# Issue Checklist
needs_update = np111.difference(np112)
for package_name in sorted(needs_update):
    print( " - [ ] [{name}](https://github.com/conda-forge/{name}-feedstock)".format(name=package_name))
jjhelmus commented 7 years ago

Without a Windows NumPy 1.12 package it may not make sense to rerender these feedstocks to add support for NumPy 1.12. Once a Windows packages is available the feedstock would need to be rerendered a second time. Perhaps waiting for short time for a Windows package is a better idea so that only a single rerendering is required.

ChrisBarker-NOAA commented 7 years ago

On Tue, Jan 17, 2017 at 10:57 AM, Jonathan J. Helmus < notifications@github.com> wrote:

Without a Windows NumPy 1.12 package it may not make sense to rerender these feedstocks to add support for NumPy 1.12. Once a Windows packages is available the feedstock would need to be rerendered a second time. Perhaps waiting for short time for a Windows package is a better idea so that only a single re rendering is required.

Agreed.

What is the hold up with the Windows Build ?

Do we need to wait for it to show up in defaults?

-CHB

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/conda-forge.github.io/issues/313#issuecomment-273263999, or mute the thread https://github.com/notifications/unsubscribe-auth/AA38YA2ztsPaqnPqEzxACdAjw3uSKw7Xks5rTQ83gaJpZM4LmAe- .

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

jjhelmus commented 7 years ago

I think the lack of a good open source BLAS implementation on Windows is the hold up. From what I recall OpenBLAS does not compile well with Visual C++ on Windows. Perhaps ATLAS is an option although compiling that on Windows is also tricky.

I opened conda-forge/numpy-feedstock#46 to track Windows builds of NumPy.

ChrisBarker-NOAA commented 7 years ago

I think the lack of a good open source BLAS implementation on Windows is the hold up.

Yup :-( -- maybe soon? But we've been saying that for a good while yet.

From what I recall OpenBLAS does not compile well with Visual C++ on Windows.

I think it needs Fortran, actually :-(

So does Vonda-forge on Windows depend on the mkl numpy in defaults?

Maybe a not-so-good performing bumpy would be an OK option.

-CHB

astaric commented 7 years ago

May I suggest an update for the last two lines of your script

for package_name in sorted(np111):
    print( " - [{updated}] [{name}](https://github.com/conda-forge/{name}-feedstock)".format(name=package_name, updated="x" if package_name in np112 else " "))

The updated version lists all relevant packages and ticks the checkbox for those that have already been updated (instead of skipping them).

kain88-de commented 7 years ago

I've looked into the order the updates need to happen to avoid dependency conflicts. In the lists below every package in Round 1 can be safely updated while for a package in Round 2 depends on other packages in Round 1. This should help finding out which packages should be prioritized in updating to numpy 1.12. I uploaded a gist with the code to find this.

Update Round 1

grlee77 commented 7 years ago

More than half of these don't have windows builds and can be updated now. I did a couple of them (scikit-umfpack and pynfft). scikit-sparse ran into a couple of test failures on numpy 1.12 that will need to be sorted out.

numpy x.x packages without windows builds

The list above was created by appending the following lines to the previously posted script by @jjhelmus

url = 'https://conda.anaconda.org/conda-forge/win-64/repodata.json'
response = urlopen(url)
cf_index_win64 = json.loads(response.read().decode('utf8'))
pkgs_win64 = cf_index_win64['packages']

url = 'https://conda.anaconda.org/conda-forge/win-32/repodata.json'
response = urlopen(url)
cf_index_win32 = json.loads(response.read().decode('utf8'))
pkgs_win32 = cf_index_win32['packages']

def split_win(names, pkgs_win64, pkgs_win32):
    needs_update_nowin = []
    needs_update_win = []
    for name in names:
        nmatch = len([p for p in pkgs_win64.values() if p['name'] == name])
        nmatch += len([p for p in pkgs_win32.values() if p['name'] == name])
        if nmatch == 0:
            needs_update_nowin.append(name)
        else:
            needs_update_win.append(name)
    return sorted(needs_update_nowin), sorted(needs_update_win)

np111_nowin, np111_win = split_win(np111, pkgs_win64, pkgs_win32)
needs_update_nowin, needs_update_win = split_win(needs_update, pkgs_win64, pkgs_win32)

for package_name in sorted(np111_nowin):
    print( " - [{updated}] [{name}](https://github.com/conda-forge/{name}-feedstock)".format(name=package_name, updated="x" if package_name not in needs_update_nowin else " "))
kain88-de commented 7 years ago

How can there be packages like astropy released on windows when they depend on numpy? Do they use the defaults channel on python? Because the conda-forge channel does not have any release of numpy on windows.

ocefpaf commented 7 years ago

They fall back to defaults on that case.

On Feb 9, 2017 6:29 PM, "Max Linke" notifications@github.com wrote:

How can there be packages like astropy https://anaconda.org/conda-forge/astropy/files released on windows when they depend on numpy? Do they use the defaults channel on python? Because the conda-forge channel does not have any release of numpy on windows.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/conda-forge/conda-forge.github.io/issues/313#issuecomment-278779742, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6BL7wgnIwwA8E5F7j9wf89ZevDYS0gks5ra4VOgaJpZM4LmAe- .

kain88-de commented 7 years ago

So all packages with windows build will have to wait until the default channel upgrades numpy for the update? Otherwise they need to be rebuild on windows again later.

ocefpaf commented 7 years ago

Unfortunately yes.

jakirkham commented 7 years ago

Feel free to contribute to the discussion in issue ( https://github.com/conda-forge/numpy-feedstock/issues/46 ), @kain88-de.

kimwalisch commented 7 years ago

@jjhelmus

NumPy 1.12 was recently released. Packages that make use of the NumPy C API need to be built against a given minor NumPy version. In order for these packages to support NumPy 1.12, they must be rebuilt against NumPy 1.12.

I am the current maintainer of primesieve-python which links against the NumPy C API. I am not a python expert so can you please explain why upon the release of the new NumPy 1.12 version primesieve-python (or python-primesieve) needs to be rebuilt and linked against NumPy 1.12? Isn't the NumPy 1.12 C API backwards compatible with version 1.11?

bsipocz commented 7 years ago

The windows 1.12 builds are out on the default channels, so I guess the updates can be started for all groups. I'm not exactly sure how this works, will there be automated PRs to all the feedstock, or should it be individual PRs? E.g. I would need the astropy updates for another channel, it's in group 1 above so ideally can be done already.

jakirkham commented 7 years ago

Hi @kimwalisch, I'm maintaining the conda-forge package for primesieve-python.

As far as NumPy goes, this is how conda/conda-build decided to handle packages that link to NumPy's C-API was by pinning major and minor version used for the build as a run time (install time) requirement. The NumPy C-API can change between minor releases and sometimes does. Take this from the 1.11.0 release for example where they changed the PyUFuncObject struct. I don't notice any similar breaks in 1.12.0, but there are some additions. FWICT it seems NumPy tries to go through deprecation cycles for these sorts of changes, but they may end up breaking the C-API in a minor release. Even though not all releases require pinning the major and minor version, it does reduce the maintenance burden of packagers and reduce the chances of a bad version constraint. IMHO the conda/conda-build developers at the time did the right thing by setting up numpy to pin the major and minor versions. Maybe someone closer to the NumPy projects development cycle can clarify how this works.

Side note: If you are interested in having permissions to the python-primesieve-feedstock, I'll happily add you.

kimwalisch commented 7 years ago

@jakirkham

Side note: If you are interested in having permissions to the python-primesieve-feedstock, I'll happily add you.

Yes, please!

jakirkham commented 7 years ago

Side note: If you are interested in having permissions to the python-primesieve-feedstock, I'll happily add you.

Yes, please!

Please see PR ( https://github.com/conda-forge/python-primesieve-feedstock/pull/7 ).

jakirkham commented 7 years ago

Well NumPy 1.13 is out. I hope we have finished this. If not, maybe it is time to start pinging people.

kain88-de commented 7 years ago

@jakirkham I've run the script from @jjhelmus again today. These are all the packages that still came up.

jakirkham commented 6 years ago

Is this still relevant? Asking in part because the new NumPy build strategy changes things significantly. Admittedly there may still be packages that have not adopted in which case we need a new listing.

jjhelmus commented 6 years ago

There may be a few packaages that still lack NumPy 1.12 (or 1.13, 1.14) support but I don't think it makes sense to track them here. Using the NumPy build strategy removes the need to rebuild packages when a new minor NumPy release is made.