search

conda-forge / cramjam-feedstock

A conda-smithy repository for cramjam.
BSD 3-Clause "New" or "Revised" License
0 stars 6 forks source link

Arch Migrator #14

Closed regro-cf-autotick-bot closed 3 years ago

regro-cf-autotick-bot commented 3 years ago

This feedstock is being rebuilt as part of the aarch64/ppc64le migration.

Feel free to merge the PR if CI is all green, but please don't close it without reaching out the the ARM migrators first at @conda-forge/arm-arch.

If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase code>@<space/conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. If you would like a local version of this bot, you might consider using rever. Rever is a tool for automating software releases and forms the backbone of the bot's conda-forge PRing capability. Rever is both conda (conda install -c conda-forge rever) and pip (pip install re-ver) installable. Finally, feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/autotick-bot/actions/runs/833321005, please use this URL for debugging

conda-forge-linter commented 3 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

milesgranger commented 3 years ago

Getting ERROR: cramjam-2.3.0-cp38-cp38-manylinux_2_24_ppc64le.whl is not a supported wheel on this platform. for all py36 thru py39; not sure why. Will check back later when I have more time

EDIT: This only applies to the ppc64le arch; all other platforms pass as usual.

milesgranger commented 3 years ago

The install works fine in the cramjam CI; probably I'm missing something obvious.

@conda-forge/arm-arch any pointers?

hmaarrfk commented 3 years ago

Why are you installing from source and installing pre built wheels

milesgranger commented 3 years ago

Because the build system is a bit complex for wheels pushed to PyPi and I'm not interested in maintaining two build systems.

Is there a better way to install right from pypi/wheels?

hmaarrfk commented 3 years ago

Unfortunately, i believe conda-forge recommends maintaining the meta.yaml as a parallel build system.

I feared that you were trying to avoid recompiling a package, but this is where pypi and conda start to differ.

conda-forge has has alot of support from people interested in alteranative computer architectures and thus ppc64le and aarch.

I remember there was a package genrator that would generate the package for you automatically. I just stopped using it since it had limited value for complicated packages.

If you want your users to have a smooth installation path, you should rebuild your package from source for conda-forge. (even if you don't care about supporting ppc64le).

milesgranger commented 3 years ago

conda-forge has has alot of support from people interested in alteranative computer architectures and thus ppc64le and aarch.

I'm not sure what you mean, I already support these archs from PyPi as it is

Isn't it easier and more developer-friendly just to support the defacto distribution for Python packages? Otherwise, users of conda can already use pip for wheels so I'm really struggling to see why I should invest time in maintaining two pretty complex build systems.

milesgranger commented 3 years ago

@martindurant you're the only one I know that uses cramjam via conda. I don't have the time to maintain two different build systems which strive for basically the same output from source.

Is it an option to have fastparquet depend on cramjam via pip? Seems like it's supported in the environment.yml files with a pip section

Or maybe continue as-is, just ignoring ppc64le?

martindurant commented 3 years ago

Is it an option to have fastparquet depend on cramjam via pip?

This could be something we advise people to do, but not something that can be described for the fastparquet conda package itself :|

just ignoring ppc64le

I am fine with this. For fastparquet 0.6.0, there were 33 conda downloads for all PPC versus, for example, 300 for linux-64-py37. The few PPC users can choose to get the wheel.

milesgranger commented 3 years ago

Okay, I'm going to try one last thing, I think it's just the naming of the wheel that is wrong, and that's why it passes on my CI but not here. Will check back. :+1:

hmaarrfk commented 3 years ago

I think it is OK to skip ppc64le.

Does this depend on maturin when installed?

hmaarrfk commented 3 years ago

I'm really struggling to see why I should invest time in maintaining two pretty complex build systems.

I understand that value might not be directly apparent as to why you should support parallel build system. I'm not trying to convince you that you should, you probably should not if the current way of installing things is what your users want and need.

I'm just explaining why your builds failed. As somebody who doesn't depend on cramjam, I'm not super invested in "fixing" what I consider to be non-compliant build, but rather pointing it out when I help was requested.

https://www.youtube.com/watch?v=Kamld5Z-xx0

^^^^^ FYI: this video explains why conda packages are important, and how they help people build systems that have dependencies beyond what pip can install (i.e. just python packages). But again, this is just a technology, and if it doesn't matter to your users, you shouldn't feel pressured to buy into it.

In terms of supporting many architectures, you have done quite a good job. You do support universal2 osx builds. Cool!

martindurant commented 3 years ago

Unfortunately for cramjam, fastparquet does benefit a lot from conda packaging! For example, the recent ABI breakage in numpy 1.20 means that some fraction of users cannot run wheels, but this doesn't happen with conda. Their only recourse is to install from source, needing a complete local build tool-chain, and optional arguments to pip. That's certainly dis-incentive enough.

(until the latest version, fastparquet also required numba, which has been very hard to manage as wheels, since it needs llvm at runtime and close coupling with python versions; and isn't available on some platforms)

milesgranger commented 3 years ago

Does this depend on maturin when installed?

No, only for development/building - cramjam has no dependencies when installed.

but okay, thank you both for the pointers; my hope was these two technologies (conda and pypi) would not be mutually exclusive but could play well together to avoid repeat work (or close to repeat) but I see that might not work so well in practice.

I'll reconsider if I can spend some time getting this to work more natively to conda, but I'm on paternity leave now and otherwise tight for time as it is.

hmaarrfk commented 3 years ago

Do enjoy your paternity leave.

conda-forge encourages collaboration, so you can definitely leave the arch migration for people that are interested in getting it working while you enjoy your time with your family member!!!