Closed Shrews closed 2 months ago
Here is a list of related issues:
Fixes:
Impacts:
Hi, thanks for working on this!
Sorry for adding a comment before making enough test, but since this PR at this moment simply uses \n
to separate lines for requirements.txt
for collections (in pip_file_data
) and process them line by line, I wonder if the collections' requirements.txt
would behave unexpectedly if it contained line continuations with backslashes.
I wonder if the collections' requirements.txt would behave unexpectedly if it contained line continuations with backslashes
@kurokobo that is correct. We discussed this when architecting the functionality, and should a package that is excluded have a backslash with a newline, it will likely cause problems.
We don't plan to add any specific functionality to permit or deny newlines, but we do intend to add documentation that states that it is in the best interest of everyone for collection maintainers, and EE creators to use pep508 formatted requirements files. We may be going so far to add linting rules, potentially via galaxy-importer
to warn when a collection defines non-pep508 formatted requirements files.
@sivel Thanks for providing detailed background! I have no objection to the policy, but I want to know details a bit.
If we strictly follow pep508, even any comments (#
) would not be allowed. But this PR does not remove an implementation that intentionally ignores comments; is it a policy that requires strictly forrowing pep508, but allows comments as an exception? Or is my understanding incorrect?
F.Y.I., to support line continuations, we need just one modification (I don't think it will be adopted, but I'll write it down as I think of it😅 ).
diff --git a/src/ansible_builder/_target_scripts/introspect.py b/src/ansible_builder/_target_scripts/introspect.py
index 553cf81..c32d121 100644
--- a/src/ansible_builder/_target_scripts/introspect.py
+++ b/src/ansible_builder/_target_scripts/introspect.py
@@ -26,7 +26,7 @@ def read_req_file(path):
def pip_file_data(path):
- pip_content = read_req_file(path)
+ pip_content = read_req_file(path).replace('\\\n', '')
pip_lines = []
for line in pip_content.split('\n'):
@sivel Thanks for providing detailed background! I have no objection to the policy, but I want to know details a bit.
If we strictly follow pep508, even any comments (
#
) would not be allowed. But this PR does not remove an implementation that intentionally ignores comments; is it a policy that requires strictly forrowing pep508, but allows comments as an exception? Or is my understanding incorrect?
Comments will be allowed. Any non-pep508 compliant entries are going to be passed through without change, unchecked and unverified. We plan to update the docs that using such non-compliant entries will be considered undefined/unsupported behavior.
Removed association with issue #364. This PR would pass through unrecognized (and potentially invalid) requirements.
Hi there, unfortunately #664 appears to be breaking numerous previously known-good custom EE builds that I am seeing in a variety of different user environments recently, even those which had already been migrated to v3 spec files, etc. While I'm sure it was well-intentioned, if even a relatively simple custom EE for something like Palo Alto will no longer build if the paloaltonetworks.panos collection itself is given in its galaxy requirements alongside its underlying python requirements (pan-python and pan-os-python), which certainly appears to be the case in ansible-builder 3.1.0, then it probably should be reviewed.
Based on the changelog, it seems like it was anticipated that it might "cause build issues in images with older versions of pip that cannot handle duplicate requirement entries". However, even on pip latest, this change (to expectations for the Ansible Builder community on how any duplicate Python dependencies will be handled and resolved at build time) is breaking things, and evidently without any definitive, prescriptive guidance so far on what exactly everyone should do going forward.
(ab300) $ pip3 --version
pip 24.2 from /Users/x/ab300/lib/python3.11/site-packages/pip (python 3.11)
(ab300) $ ansible-builder build -t palo-ee
Running command:
podman build -f context/Containerfile -t palo-ee context
Complete! The build context can be found at: /Users/x/palo-ee/context
(ab300) $ ansible-builder --version
3.0.0
(ab300) $ deactivate
$ source ab310/bin/activate
(ab310) $ ansible-builder --version
3.1.0
(ab310) $ ansible-builder build -t palo-ee
Running command:
podman build -f context/Containerfile -t palo-ee context
...showing last 20 lines of output...
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ PS1='(venv) '
++ export PS1
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ '[' -f /tmp/src/upper-constraints.txt ']'
+ [[ -n '' ]]
+ install_wheels
+ '[' -f /tmp/src/build-requirements.txt ']'
+ '[' -f setup.py ']'
+ '[' -f /tmp/src/requirements.txt ']'
+ '[' '!' -f /output/requirements.txt ']'
+ /usr/bin/python3 -m pip install --cache-dir=/output/wheels -r /tmp/src/requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
ERROR: Double requirement given: pan-os-python (from -r /tmp/src/requirements.txt (line 13)) (already in pan-os-python==1.8.0 (from -r /tmp/src/requirements.txt (line 1)), name='pan-os-python')
Error: building at STEP "RUN /output/scripts/assemble": while running runtime: exit status 1
An error occurred (rc=1), see output line(s) above for details.
(ab310) $
Hello @djgoosen. In the future, it's better to open a new issue than to comment on a closed PR since we may not always notice these types of comments.
The 3.1
release was sort of a big update with some very necessary changes that were required in order to progress the project forward and eliminate some unmaintained external libraries. Backward compatibility with EEs that work with version 3.0
was never guaranteed, thus the new Y-release version.
In your particular case, upgrading pip
within your EE will likely fix your problem. In your PR description, you are showing the pip
version information for software outside of the container image, but it is the version of pip
within the container image that matters (that is where Python packages are installed). You can either use a more recent base image with a newer pip
, or use additional_build_steps
to upgrade the version of pip
within the container image.
If that does not resolve your issue, you also have the option of using the new exclude
feature in 3.1
to ignore any dependencies from collections that may be causing you issues.
If both of those solutions fail for you, please open a new issue and provide the EE file where you are seeing the problem.
Based on #647
Changes
requirements_parser
Python dependency.--sanitize
option still exists, but is hidden, and it does nothing.EXCLUDE_REQUIREMENTS
only applied to pip before, simply because sanitize_requirements was only ever used for pip, but it now applies to system as well.requirements.txt
entries are simply pushed down topip
(-r
still causes recursive inclusion inline).exclude
schema keyword is added to version3
EE format and requiresansible-builder
version3.1
or higher. Using an older builder version simply errors.Bug Fixes
#
anchor from requirement URLs (e.g.,git+https://git.repo/some_pkg.git#egg=SomePackage
)Example
TODO:
all_from_collection
orfrom_collection
?)