Uncouple from `pip` a bit more

jaimergp commented 2 months ago

There are libraries we can use for better integration instead of relying so much on pip subprocesses. This could be part of a refactor where we isolate the components of the package a bit better.

Right now we have, roughly:

A CLI that mimics a subset of pip's UX with a couple extra flags
A PyPI solver backend based on: grayskull, pip; uv is planned too
A conda/pypi mapper based on several public resources (cf-graph-countyfair, grayskull, parselmouth)
A PyPI installer that essentially builds the command to be passed to pip. This could be reimplemented using PyPA's build and installer so conda can provide the build-system.requirements (which means we can offer compilers). This would also allow us to inject the right kind of metadata in the *.dist-info directories, in the right way.

cc @dholth

xref:

jezdez commented 2 months ago

The PyPA build/installer integration seems backwards to me, as this package is about enabling conda users to successfully use Python packages from PyPI in their conda environments, not build new Python wheels that would be better served by conda packages in the first place.

Of course, it'd be splendid if the .conda file format would simply be a hypothetical .whl format plus metadata extensions. But then we're kind of back at square one with conda-press (talk).

jaimergp commented 2 months ago

This is only for the cases where we didn't find the corresponding conda artifact and the user insists on serving something from PyPI-compatible sources.

If there's only a sdist (or a sdist-compatible repository / directory), pip will build a wheel anyway. What happens behind that subprocess is underspecified and pip will create the build environment on its own with no conda involvement.

The intent here would be to have some control over that build environment; this is particularly important for those packages that require compilation so we provide the compilers and whatnot in an ABI compatible way.

Although at that point we could also just say "grayskull this and run conda-build on top", and that's also a valid option (albeit with its overheads). It would be one more option, I guess?

I don't see this as an immediate priority but I wanted to capture Daniel's thoughts before getting them lost in the sea of merged PRs and closed issues.

dholth commented 2 months ago

Other installers are also trying to perfect the integration of conda and pypi package, I haven't researched those yet. I'm imagining something like

$ conda pypi-build-install-develop --editable <path/to/somepackage> or <[pep 508](https://peps.python.org/pep-0508/) requirement>
<installs enscons or flit or whatever from conda packages, gets metadata>
[y/N] Accept solution?
channels:
  - defaults
dependencies:
    - aiofiles
    - aiohttp
    - click
    - pip:
        - json_log_formatter
        - ...
<option to adjust where each package comes from with a configuration file>
Y
<it is installed>

As Jaime says, we gain control over installing the pyproject build-requires from a conda source and we also gain control over the the environment used to build the pypi package.

Possibly an automatic build of a whole tree of packages in the same way that pip wheel populates a directory with all the wheels needed to install something, after which you can install offline from that directory of wheels. I'm less clear on how this would be designed compared to the simpler "conda develop" replacement.

conda-incubator / conda-pypi

Uncouple from `pip` a bit more #37