Support for pip install --build-options / --global-options / --install-options

[pmav99]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

There are some packages that in order to be installed you need to pass additional options to setup.py. There is support for this in pip. E.g. to install GDAL on Ubuntu 18.04 you need to install gdal-bin and libgdal-dev using apt and then to issue:

pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`

The question is, does poetry support this? Is there a way to specify these options in pyproject.toml? If not, how do you suggest handling this case in a project that depends on such packages?

[pakal]

I confirm that without such options, some packages like Pillow, rcssmin etc. can be a pain to install, especially when they try to build optional C extensions on environments where it's not possible. Having these pip-like options to be put in the listing of dependencies would be a great feature.

[higherorderfunctor]

I would also like to see this supported in native poetry projects so the options are passed to the undocumented build property for building C extensions.

Ideally I could have one poetry project that builds a C extension. Then in another poetry project, pass the build options.

[tool.poetry.dependencies]
python = "^3.7"
other-project = { git = "ssh://.../other-project.git", branch = "master", install-option="--some-lib-prefix=/opt/some-lib" }

[pmav99]

Could someone provide some hint on how to tackle this? Where would we need to make changes in order to allow for the syntax proposed by @higherorderfunctor ?

[higherorderfunctor]

I did some preliminary exploring. Assuming each dependency is installed via a different call to PIP, then those args would need to be passed at this line. https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L115

There are two paths that define *args that can be found here and here: https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L41 https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L182

The entry point for these paths is here and here. I am assuming those options would need to be extracted from the package argument and included in *args: https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L30 https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/pip_installer.py#L170

The toml schema for dependencies is here which I assume needs to be aware of the two options: https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/json/schemas/poetry-schema.json#L193

The dependency definition that I assume is passed to the install commands above is located here which would need to be aware of the options: https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/packages/dependency.py#L21

Dependencies are built here which looks to parse the options out of the toml file and build the dependency definition which would need to be updated: https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/packages/package.py#L264

I haven't fully traced the life-cycle of a Dependency to the PipInstaller. It looks like Dependencies are stored on the package here (I don't think any code change would be needed here): https://github.com/sdispater/poetry/blob/master/poetry/packages/package.py#L68

Dependencies look to be "converted" to packages here (no code change needed I can see): https://github.com/sdispater/poetry/blob/b218969107e49dc57e65dbc0d349e83cbe1f44a8/poetry/installation/installer.py#L495

More maybe needed... such as ensuring those options are exposed post-build.

[jkyl]

@higherorderfunctor thanks for the exploration, that is very helpful. I am interested in something quite similar: passing a link to pip install --find-links. This would enable me to pull the appropriate .whl for pytorch on Windows (pytorch's pypi only contains linux/mac .whls).

[jkyl]

https://github.com/sdispater/poetry/pull/1511

[reedjosh]

https://github.com/python-poetry/poetry/pull/2017

[Matesanz]

Thanks for the Feature! PR says documentation on this was updated but I'm not able to find it, could you please provide the link or an example to this feature? Thanks in advance.

[reedjosh]

Hey, sorry. I think I just left the tic-boxes checked when I submitted the template https://github.com/python-poetry/poetry/pull/2017

I did not add tests or documentation. I would be happy to add these items, but I wanted to verify that the PR was using acceptable methods before moving forward.

I will continue the discussion there.

[davidbeers]

Lack of support for these pip options is the main thing that holds me back from adopting Poetry right now. Like @pmav99 I need the gdal package and it can't be installed properly without the mentioned options.

[abn]

@davidbeers is it correct to assume that the build options can vary depending on user installing the project? As in while some users might use -I/usr/include/gdal others might need another path based on their distro/platform?

[abn]

Also, looking at the original request, I suspect a build script should do the trick here. See https://github.com/python-poetry/poetry/issues/5539#issuecomment-1119047694 https://github.com/python-poetry/poetry/issues/5539#issuecomment-1126818974.

[davidbeers]

is it correct to assume that the build options can vary depending on user installing the project? As in while some users might use -I/usr/include/gdal others might need another path based on their distro/platform?

I imagine that's true since this path is based on a Debian installation of the native libgdal-dev library. A different distro might mean a different path.

Also, looking at the original request, I suspect a build script should do the trick here.

So is the idea that the build script would from a subprocess run pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL=='gdal-config --version'?

[abn]

The idea is that the you can add your own custom build.py that will get executed by poetry-core based on your configuration in pyproject.toml.

When pip builds the project, it calls poetry-core. Then poetry-core will run the build script as python build.py. This happens within an environment with all the build dependencies of your project as defined in build-system.requirements are installed. So in theory you could have:

def build():
    # call subprocess("gcc -I/usr/include/gdal") or something
    pass

if __name__ == "__main__":
    build()

[davidbeers]

Thanks, this does seem to be a reasonable workaround. It means that when installing libraries with pip install options the changes in those options or in a pinned version will not be detected by Poetry when calling poetry update but I can handle that for one or two libraries that have special install options for now. Still it would still be great to support a proposal like what @higherorderfunctor suggested to expose the install options functionality of pip.

[mmathys]

This would be helpful for me as well

[eliasmistler]

Same issue here - I can't install pygraphviz through poetry (on MacOS with M1/M2) without this (see https://stackoverflow.com/questions/40266604/pip-install-pygraphviz-fails-failed-building-wheel-for-pygraphviz and https://stackoverflow.com/questions/15661384/python-does-not-see-pygraphviz/39976362#39976362)

[david-waterworth]

I'm having the same issue as @eliasmistler with graphviz on my macbook pro. The official (pip-based) solution is https://pygraphviz.github.io/documentation/stable/install.html#homebrew

pip install --config-settings="--global-option=build_ext" \
            --config-settings="--global-option=-I$(brew --prefix graphviz)/include/" \
            --config-settings="--global-option=-L$(brew --prefix graphviz)/lib/" \
            pygraphviz