search

poise / poise-python

A Chef cookbook to provide a unified interface for installing Python, managing Python packages, and creating virtualenvs.
Apache License 2.0
124 stars 108 forks source link

Can't install packages/wheels that consist of underscores #123

Closed mattdodge closed 6 years ago

mattdodge commented 6 years ago

Related to (caused by?) #71.

By substituting all underscores with dashes (https://github.com/poise/poise-python/commit/73b0999ca279191b2909aaaeb858282d1438e3f4) it makes it impossible to install wheel files (or packages from local repositories) that consist of underscores.

For example, if I build a local packaged wheel file with package name my_package then the resulting wheel filename will be my_package-1.0.0-py3-none-any.whl. Then running a recipe that looks like this:

python_package 'underscore_wheel' do
 package_name 'my_package-1.0.0-py3-none-any.whl'
end

will give you some failure logs like this:

Requirement 'my-package-1.0.0-py3-none-any.whl' looks like a filename, but the file does not exist

You can see that the underscore was replaced with a hyphen and thus it can't find the wheel. I know PyPI used to (don't know if it still does) prevent underscores in their package names but if the python_package resource is intended to be used for more than just PyPI packages (which I assume it is) then I don't think it makes sense to carry that replacement into this codebase.

coderanger commented 6 years ago

So python_package only works with "normal" packages by design. It doesn't support stuff like URLs or filenames. I don't think 73b0999 is actually related (it only applies to the internal representation, not what gets passed to pip), but if this works at all with non-underscore wheel filenames, I promise you that was an accident.

The problem with all the non-"normal" package name strings that pip supports is that Chef would have no way to be idempotent in that situation, it dosn't know how to get the current version of the package or check if the wheel is a different version. pip itself knows how to do this, but it doesn't expose enough info to script that from the outside. If you are okay with non-idempotent behavior (read: it will re-run the install every converge) or want to control idempotence yourself via notifications or guard statements, you can use this:

python_execute '-m pip install my_package-1.0.0-py3-none-any.whl'
mattdodge commented 6 years ago

Yeah I have it wired up right now that way, with the python_execute. The downside is exactly what you described, it will run this every time. The executions at least are fast if nothing changed, but that's a bummer that it's not possible (at least easily possible) to bring that idempotence in to chef here.

Thanks for the quick response though, I'll close this now as it seems like the desired behavior for wheel files is to use python_execute.

coderanger commented 6 years ago

If you want to help petition Pip for a "dry run" mode of some kind, I could some day support that kind of thing. I already had to implement a terrible hack just to do the data gathering for normal packages >_<