Closed carlcsaposs-canonical closed 1 day ago
sed-i
commented
1 year ago I imagine this is by design: Lib authors should avoid pinning in PYDEPS if possible. If they do pin, it should be for a good reason. By having PYDEPS at a higher priority, we're more likely to have things fail earlier, which is a good thing.
carlcsaposs-canonical
commented
1 year ago I imagine this is by design: Lib authors should avoid pinning in PYDEPS if possible. If they do pin, it should be for a good reason. By having PYDEPS at a higher priority, we're more likely to have things fail earlier, which is a good thing.
The problem is that unpinned PYDEPS will override pinned deps in requirements.txt
lengau
commented
1 year ago I realised my thoughts about this have been scattered, so I'm writing this comment just to put everything into a coherent bundle.
The current process works as follows:
--upgrade--upgrade--upgrade--upgrade (<-- our antagonist)In the general case, this might be worth reconsidering as a whole (see #1140). But in the more specific case of this bug, we probably just need to ensure that charmlib deps don't override requirements.txt.
The best way to do this is probably to combine steps 3 and 4 into a single pip command. Step 2 can probably come along with that essentially for free, but step 1 will need its own thoughts.
lengau
commented
12 months ago Closing to be merged into #1135
carlcsaposs-canonical
commented
11 months ago Closing to be merged into #1135
@lengau which issue/pr was this merged into? #1135 is this issue
DnPlas
commented
2 months ago Hi @lengau, I understand that this issue should've been resolved by #1157, but I am still running into it. Interestingly enough, there are a number of inconsistencies with how the packages get installed and how their versions get overridden. Here are my findings:
charmcraft init.requirements.txt only pins ops ~= 2.5charmcraft.yamlname: my-charm
type: charm
title: Charm Template
summary: A very short one-line summary of the charm.
description: |
A single sentence that says what the charm is, concisely and memorably.
A paragraph of one to three short sentences, that describe what the charm does.
A third paragraph that explains what need the charm meets.
Finally, a paragraph that describes whom the charm is useful for.
bases:
- build-on:
- name: ubuntu
channel: "20.04" <-- please NOTE I have used this and newer versions like 22.04
run-on:
- name: ubuntu
channel: "20.04"
containers:
httpbin:
resource: httpbin-image
resources:
httpbin-image:
type: oci-image
description: OCI image for httpbin
upstream-source: kennethreitz/httpbinTake the charm from before and do the following:
PYDEPS, e.g. charmcraft fetch-lib charms.prometheus_k8s.v0.prometheus_scrapecosl==0.0.12 in requirements.txt:cosl==0.0.12
ops ~= 2.5cosl was as expected::: :: Successfully installed cosl-0.0.12 ops-2.15.0 pyyaml-6.0.1 typing-extensions-4.12.2 websocket-client-1.8.0 wheel-0.43.0 requirements.txt - wget https://raw.githubusercontent.com/canonical/knative-operators/track/1.12/charms/knative-operator/requirements.txt$ charmcraft clean
Cleaning project 'my-charm'.
Cleaned project 'my-charm'.
$ charmcraft pack -vcosl gets installed with the expected version, but then upgraded at some point during the build and the pin gets ignored. Expected is 0.0.12.:: :: Installing collected packages: lightkube-models, zipp, websocket-client, urllib3, typing-extensions, tenacity, sniffio, ruamel-yaml-clib, pyyaml, pyrsistent, pkgutil-resolve-name, ordered-set, markupsafe, idna, h11, exceptiongroup, charset-normalizer, certifi, attrs, ruamel-yaml, requests, ops, jinja2, importlib-resources, deepdiff, anyio, jsonschema, httpcore, cosl, serialized-data-interface, httpx, lightkube, charmed-kubeflow-chisme
:: :: Successfully installed anyio-4.0.0 attrs-23.1.0 certifi-2023.7.22 charmed-kubeflow-chisme-0.2.0 charset-normalizer-3.2.0 cosl-0.0.12 deepdiff-6.2.1 exceptiongroup-1.1.3 h11-0.14.0 httpcore-0.17.3 httpx-0.24.1 idna-3.4 importlib-resources-6.0.1 jinja2-3.1.2 jsonschema-4.17.3 lightkube-0.14.0 lightkube-models-1.28.1.4 markupsafe-2.1.3 ops-2.15.0 ordered-set-4.1.0 pkgutil-resolve-name-1.3.10 pyrsistent-0.19.3 pyyaml-6.0.1 requests-2.31.0 ruamel-yaml-0.17.32 ruamel-yaml-clib-0.2.7 serialized-data-interface-0.7.0 sniffio-1.3.0 tenacity-8.2.3 typing-extensions-4.11.0 urllib3-2.0.4 websocket-client-1.6.2 zipp-3.16.2
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', 'cosl']
:: :: Requirement already satisfied: cosl in ./staging-venv/lib/python3.8/site-packages (0.0.12)
:: :: Collecting cosl
:: :: Using cached cosl-0.0.15-py3-none-any.whl
:: :: Requirement already satisfied: ops in ./staging-venv/lib/python3.8/site-packages (from cosl) (2.15.0) This case is even weirder, as it seems like charm-binary-python-packages is affecting the behaviour of installs.
requirements.txt from case 2charmcraft.yaml...
bases:
- build-on:
- name: ubuntu
channel: "20.04"
run-on:
- name: ubuntu
channel: "20.04"
parts:
charm:
charm-python-packages: [setuptools, pip]
charm-binary-python-packages: [jinja2]
...$ charmcraft clean
Cleaning project 'my-charm'.
Cleaned project 'my-charm'.
$ charmcraft pack -v:: :: Successfully installed anyio-4.0.0 attrs-23.1.0 certifi-2023.7.22 charmed-kubeflow-chisme-0.2.0 charset-normalizer-3.2.0 cosl-0.0.12 deepdiff-6.2.1 exceptiongroup-1.1.3 h11-0.14.0 httpcore-0.17.3 httpx-0.24.1 idna-3.4 importlib-resources-6.0.1 jinja2-3.1.2 jsonschema-4.17.3 lightkube-0.14.0 lightkube-models-1.28.1.4 markupsafe-2.1.3 ops-2.14.0 ordered-set-4.1.0 pkgutil-resolve-name-1.3.10 pyrsistent-0.19.3 pyyaml-6.0.1 requests-2.31.0 ruamel-yaml ruamel-yaml-clib serialized-data-interface-0.7.0 sniffio-1.3.0 tenacity-8.2.3 typing-extensions-4.11.0 urllib3-2.0.4 websocket-client-1.6.2 wheel-0.43.0 zipp-3.16.2 Do exactly the same as case 3, but the part in charmcraft.yaml will look like this:
parts:
charm:
charm-python-packages: [setuptools, pip]
charm-binary-python-packages: [sh] <--- this is a different package Clean, build and observe the error again!
:: :: Installing collected packages: lightkube-models, zipp, websocket-client, urllib3, typing-extensions, tenacity, sniffio, ruamel-yaml-clib, pyyaml, pyrsistent, pkgutil-resolve-name, ordered-set, markupsafe, idna, h11, exceptiongroup, charset-normalizer, certifi, attrs, ruamel-yaml, requests, ops, jinja2, importlib-resources, deepdiff, anyio, jsonschema, httpcore, cosl, serialized-data-interface, httpx, lightkube, charmed-kubeflow-chisme
:: :: Successfully installed anyio-4.0.0 attrs-23.1.0 certifi-2023.7.22 charmed-kubeflow-chisme-0.2.0 charset-normalizer-3.2.0 cosl-0.0.12 deepdiff-6.2.1 exceptiongroup-1.1.3 h11-0.14.0 httpcore-0.17.3 httpx-0.24.1 idna-3.4 importlib-resources-6.0.1 jinja2-3.1.2 jsonschema-4.17.3 lightkube-0.14.0 lightkube-models-1.28.1.4 markupsafe-2.1.3 ops-2.14.0 ordered-set-4.1.0 pkgutil-resolve-name-1.3.10 pyrsistent-0.19.3 pyyaml-6.0.1 requests-2.31.0 ruamel-yaml-0.17.32 ruamel-yaml-clib-0.2.7 serialized-data-interface-0.7.0 sniffio-1.3.0 tenacity-8.2.3 typing-extensions-4.11.0 urllib3-2.0.4 websocket-client-1.6.2 zipp-3.16.2
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', 'cosl']
:: :: Requirement already satisfied: cosl in ./staging-venv/lib/python3.8/site-packages (0.0.12)
:: :: Collecting cosl
:: :: Using cached cosl-0.0.15-py3-none-any.whl
:: :: Requirement already satisfied: ops in ./staging-venv/lib/python3.8/site-packages (from cosl) (2.14.0) As suggested by @carlcsaposs-canonical, we could use charm-strict-dependencies: true in our charms to solve the issue. I did it, but now I am hitting https://github.com/canonical/charmcraft/issues/1456.
Also, most of my team's charms won't work with this because we also define charm-python-packages, so adding charm-strict-dependencies will return - 'charm-python-packages' must not be set if 'charm-strict-dependencies' is enabled in field 'parts.charm.charm-strict-dependencies'
lengau
commented
3 weeks ago I'm not sure whether we're going to be able to fix this in the charm plugin, as any changes made therein are going to break other configurations.
Instead, we're going to work to replace the charm plugin with clearer, less tightly-coupled plugins that meet people's needs.
@DnPlas - the best way to work around this for your use case is to use charm-requirements and put the requirements that were previously in charm-python-packages into a requirements.txt file. If you need multiple requirements files, you can do that by specifying both files in charm-requirements.
lengau
commented
1 day ago Marking this as not planned, as it's too complex to implement in the charm plugin. The upcoming python and poetry plugins will support this.
Bug Description
Example: pgbouncer-operator has
pydantic==1.10.9in its requirements.txt pgbouncer-operator also uses a charm lib withPYDEPS = ["cosl", "pydantic"]During
charmcraft pack, the unpinnedpydanticcharm lib dependency overrides the pinnedpydanticdependency and installs pydantic version 2To Reproduce
Environment
Ubuntu 22.04 LTS
charmcraft.yaml
Relevant log output