search

pypa / packaging

Core utilities for Python packages
https://packaging.pypa.io/
Other
614 stars 244 forks source link

Checking that a set of requirements is available and compatible (ALA pip check) #317

Open FFY00 opened 4 years ago

FFY00 commented 4 years ago

As discussed on IRC, I propose adding a requirement validator to packaging. There's is a real need for it, I need it for at least some of my projects.

The validator should find the metadata (using importlib.metadata) and validate the specified requirement string.

pradyunsg commented 4 years ago

Are you asking for https://github.com/pypa/packaging/issues/147?

FFY00 commented 4 years ago

No, sorry. I wasn't clear enough. I am asking for a way to validate if the requirements are met. This can be used for eg. in python-build to validate the build dependencies are met, or in a wheel installer, to make sure the dependencies are available, and are compatible, before installing the wheel.

FFY00 commented 4 years ago

I have a minimal implementation, but it does not validate the dependencies of extras. It would be good to have a strong implementation here, since it turns out it is not trivial.

https://github.com/FFY00/python-build/blob/master/build/__init__.py#L39

pradyunsg commented 4 years ago

I see, something like pip check but independent of pip?

FFY00 commented 4 years ago

Yes.

brettcannon commented 4 years ago

One thing to consider about this proposal is I think this would be the first bit of functionality that actually queries installed packages. Everything else in 'packaging' is mostly parsing. packaging.tags is the only thing that queries the live interpreter running, but we don't have anything that goes into the environment to accomplish something.

Now I'm not saying expanding out is bad, I just want to be upfront it does open up the scope of this project a bit (and bring in a dependency on importlib-metadata).

FFY00 commented 4 years ago

I agree, we should consider if we should use importlib.metadata or something else.

I would say the importlib-metadata dependency is just a temporary nuisance, importlib.metdata is available from Python 3.8 and up. Given time, older versions a compatible Python version will start to get adopted. If the importlib-metadata dependency is still a problem, we can declare an extra that pulls it.

pradyunsg commented 4 years ago

FWIW, I'm certainly not 100% sure if packaging is the correct place to put it. It might be that importlib should grow a helper function to do this or something.

I do think decoupling this from pip is a good idea, but IDK if packaging is the correct place.

FFY00 commented 4 years ago

But it would need to use the functionality provided by packaging. As importlib is part of the standard library, this is not possible.

I agree packaging might not be the most optimal place, but given the context, I believe it is where it makes most sense. And the functionality proposed falls somewhat in scope of the project.

Do you have any place in mind other than importlib?

brettcannon commented 4 years ago

I agree that I don't think importlib is a good fit for this.

Would this big a good fit for pradyunsg/installer?

FFY00 commented 4 years ago

I think it makes more sense to have it here. This library aims to provide "Core utilities for Python packages", I think verifying if the requirements are met is one of those workflows. pradyunsg/installer provides a library with the install logic, I do not really believe this is install logic, IMO "packaging logic" better describes it.

layday commented 3 years ago

Regardless of what happens here I think it would be useful to expose individual markers in markers.Marker so we can e.g. filter requirements based on their marker value for extras. A related issue is that Marker.evaluate will raise if its own extra is missing from the environment, so requirements cannot be evaluated if they contain an extra (i.e. if they've been extracted from wheel metadata).

FFY00 commented 3 years ago

We have a working implementation in pypa/build. How do we want to proceed?

FFY00 commented 3 years ago

Sorry to bother, but we have working code for this. Would a PR be welcome?

brettcannon commented 3 years ago

I'm still on the fence for this since it just feel like it quite fits into 'packaging' (don't ask me why; gut feel), but then I don't know where it would belong in PyPA either.

FFY00 commented 3 years ago

Yeah, and I agree that pypa/packaging might not be the best place to put it, but I think it is close enough to be relevant and from all other projects I think pypa/packaging is what makes the most sense.

I would really like to figure this out, otherwise it will be indefinitely stalled :confused:

This simply integrates the pypa/packaging functionality with stdlib's importlib.metadata.

It feels silly to create a new package just for this, but that could be done if we don't arrive to any consensus. I mean, maybe we could bundle a few package introspection routines, but I struggle to make a strong case for this new package.

layday commented 3 years ago

I suppose we could use a callback in place of importlib.metadata if that makes it more palatable for inclusion in packaging. I've adapted the implementation from build with this in mind: https://github.com/layday/build/blob/prospective-packaging-implementation/check_dependencies.py.