Closed beck3905 closed 1 year ago
Arguably, partifact
should only be concerned with configuring poetry
. We added in the pip
logic with global.index-url
as it makes sense for our unique setup internally.
Is the pip
functionality useful for you at all? I'd be tempted to make it optional through a CLI argument with the default being not configuring pip
. Alternatively, the CLI arg could control whether extra-index-url
or index-url
is set.
The pip functionality is useful for me, actually, but right now it actually is breaking me. Here is my use case.
I depend on multiple packages from the public PyPI and from my private PyPI stored in CodeArtifact. My applications are serverless applications running in AWS Lambda. So when I am developing locally, I use poetry
to manage my dependencies in my project, but I also use pip
and pipx
to manage system-wide dependencies on my local machine.
After I run partifact login
I can manage my project dependencies easily with poetry and I appreciate the simplicity partifact
provides. But then when I try to manage any system-wide dependencies, the changes to my global pip config made by partifact limit my ability to install any dependencies that aren't in my private PyPI in CodeArtifact. This is due to the global.index-url
config and should be solved by using global.extra-index-url
instead.
In addition, in my CI/CD pipeline I build Lamba layer packages. I do this by first running poetry export
to generate a requirements.txt file and then I run pip install -t ...
to install the dependencies into a local directory to be packaged. Again, due to the global.index-url
setting I cannot install dependencies other than from my CodeArtifact PyPI. So I ended up writing a command to read the global.index-url
config, set that value to the global.extra-index-url
config, and then remove the global.index-url
config.
TL;DR: pip
functionality is useful for me, but is currently broken for me. Making them optional arguments and supporting extra-index-url
instead of index-url
should help.
Thanks for the detailed write-up!
Are the packages that are not in your CodeArtifact repository in another private repository? For public packages, wouldn't having PyPI as an upstream repository for your private CodeArtifact repository suffice? Sorry, just trying to make sure I fully understand your use-case.
For the CI/CD use-case, are you exporting with --with-credentials
? Again, we only ever use a single private repository, so there may be gotchas I'm not aware of, but I feel like the exported requirements file should be self-contained without relying on the global URLs in the pip config.
If you still think setting the global.extra-index-url
is useful, my preference would be implementing this as an optional convenience through a CLI argument. What do you think?
I could have PyPI as an upstream for my private CodeArtifact, but I don't really need the extra setup to do that because I can just hit PyPI directly and my private CodeArtifact directly.
I can use --with-credentials
to export my requirements.txt and it gets added as an extra-index-url in my requirements.txt. However, the problem is still that the global.index-url
setting is limiting pip to only looking in the single private CodeArtifact repository. Poetry doesn't add PyPI to the requirements.txt as an index url or extra index url.
It seems like if I do the --with-credentials
in my export then everything I need will be in my requirements.txt as long as the global.index-url
is not set in my pip config. So I think either removing that altogether or making it optional via CLI would address my use-case. Thanks
In that case, I'd make it optional and not configure pip
by default. We can introduce a --configure-pip
flag to retain the current behaviour.
I can do it sometime this weekend. Alternatively, if you want to have a go at it, let me know.
I think that will work. I just ran a test where I run pip config unset global.index-url
and added --with-credentials
to my export and it worked. Before, without the --with-credentials
I was reading the global.index-url
value and saving it to global.extra-index-url
and then unsetting global.index-url
. So this simplifies it for me. With your proposed change I won't have to mess with pip config at all.
Unfortunately, I don't have time to work on this right now. I have a working setup so it's no big rush. I appreciate your help and responsiveness.
0.3 should address this. Any issues, let me know!
While I don't fully understand the long thread I'm linking to here, it seems germane regarding how pip
treats global.extra-index-url
: https://github.com/pypa/pip/issues/8606
This isn't a partifact
concern, but it is relevant to trying to understand pip
's approach to multiple code repositories. One important thing to note is that the entire python package namespace is a singleton. You need to give your python packages unique names. At any point, someone might publish to the pypi
repository a package with your private package's name, and, depending on how pip
resolves the package, you might get surprised if you rely on pip
's search order!
For this reason, putting CodeArtifact
"in front" of pypi
("upstream") seems to be a safer choice.
When I run
partifact login my-repo
my global pip configuration gets updated withindex-url
. Whenever I try to pip install if I have packages that are in a different index then pip can't find them.index-url
becomes the default package instead of pypi.Would it make more sense for partifact to update the
global.extra-index-url
config instead ofglobal.index-url
to allow it to work with multiple sources?