Correctly excluding a package?

[cidrbl0ck]

While this is not a bug, or I guess a real issue, I am trying to determine how to exclude a package.

The long story is: I'm needing to build a custom EE to include WinRM, PSRM, and the community.windows collection. Everything I read mentions to use the ee-supported-rhel8 as a BASE image. However building fails due to :

error: No package matches 'openshift-clients'
Error: error building at STEP "RUN assemble": error while running runtime: exit status 1

Since I don't use Openshift I'd like to exclude it. Reading here: https://access.redhat.com/solutions/6999217. it says to format your context/ContainerFile as such:

ARG EE_BASE_IMAGE=registry.redhat.io/ansible-automation-platform-21/ee-supported-rhel8
ARG EE_BUILDER_IMAGE=registry.redhat.io/ansible-automation-platform-21/ansible-builder-rhel8

FROM $EE_BASE_IMAGE as galaxy
ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
USER root

ADD _build /build
WORKDIR /build

FROM $EE_BUILDER_IMAGE as builder
ADD _build/requirements.txt requirements.txt
RUN ansible-builder introspect --sanitize --user-pip=requirements.txt --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt
RUN assemble

FROM $EE_BASE_IMAGE
ARG PKGMGR_OPTS="--nodocs --setopt=install_weak_deps=0 --setopt=rhel-8-for-x86_64-appstream-rpms.excludepkgs=ansible-core"
USER root
COPY --from=builder /output/ /output/
RUN /output/install-from-bindep && rm -rf /output/wheels

However this doesn't work as I'm sure it's a syntax problem with Ansible-Builder versus Podman. I'm an AAP shop so all I know is ansible-builder. I'm looking for information on how to exclude openshift-clients and any other un-needed package so I can build this image successfully.

[Shrews]

There is no built-in mechanism to builder to explicitly ignore a Python requirement.

I will note that the openshift-clients package doesn't seem to be a thing on PyPI: https://pypi.org/project/openshift-clients

However, the openshift-client package does exist: https://pypi.org/project/openshift-client

So looks like a spelling error in some Python requirements file from somewhere (possibly your user supplied requirements.txt from your EE, or even one of the dependencies you pull in?) If you run builder with increased verbosity (-v3 I think would do it), you should see the output of the introspect command from your Containerfile. It would list all of the coalesced Python requirements and which packages they come from. That might help you track down the misspelled package name.

[cidrbl0ck]

EDIT Apologies, I typo'd openshift-clients to openshift-client above

Thanks Shrews, Building with the highest verbosity returns this:

+ /usr/bin/microdnf install -y --nodocs --setopt install_weak_deps=0 findutils gcc libcurl-devel libssh-devel libxml2-devel make openshift-clients openssl-devel python3-rpm python38-Cython python38-devel python38-lxml python38-pytz python38-pyyaml python38-requests python39-devel python39-lxml qemu-img
error: No package matches 'openshift-clients'
Error: error building at STEP "RUN assemble": error while running runtime: exit status 1

Not super helpful but, somewhere theres a file telling microdnf to install this list of packages. Is that maybe 'embedded' in the BASE_IMAGE?

My requirements.txt doesn't exist in this case, and requirements.yml (even though prolly not related) contains 2 collections to be included.

There is an internal issue created back in mid-March with regards to the openshift-clients problem: https://access.redhat.com/solutions/6990615

Environment
Red Hat® Ansible Automation Platform 2.3
Issue
Why is ansible-builder unable to install the openshift-clients package even if it is present in the ee-supported Base Image in Ansible Automation Platform 2.3?
Resolution
This Issue is raised in our Internal Bug Tracking System and the Engineering Team is working on it.

Their workaround is to use the ee-minimal image, which I don't want to do.

[Shrews]

Ah, this is a bindep dependency, not Python, as I had assumed.

Looking at the internal issues, I don't see any suggested, good resolution. But, IMO, this does seem to be a problem with one of the RH registry images.

@Akasurde Looks like you were helping look into this on the internal issues. Any information you could supply here?

[cidrbl0ck]

I tried building again with a bindep.txt in the working dir, copy/pasted the bindep output from earlier and removed openshift-clients but the build failed.. my bindep is being ignored/ovwerwritten.

[tuxpreacher]

What does your execution-environment.yml look like? You specify the name/location of your files there.

[Akasurde]

@cidrbl0ck Also, What is the content of requirements.yml?

[cidrbl0ck]

So admittedly I did forget to put a reference to the bindep.txt file in execution-environments.yml. However correctign that and re-running the build resulted in the same thing.

---
version: 1

build_arg_defaults:
    EE_BASE_IMAGE: 'private_automation_hub/23-supported-rhel8:latest'

dependencies:
  galaxy: requirements.yml
  system: bindep.txt

Bindep.txt:

gcc
libcurl-devel
libssh-devel
libxml2-devel
make
openssl-devel
python3-rpm
python38-Cython
python38-devel
python38-lxml
python38-pytz
python38-pyyaml
python38-requests
python39-devel
python39-lxml
qemu-img

[cidrbl0ck]

Requirements.yml:

collections:
   - community.windows
   - community.general

[cidrbl0ck]

Checking the failed results further, there's another reference to openshift-clients:

• 'openshift-clients [platform:rhel-8] # from collection kubernetes.core'

But that's additional to the bindep line.. I think. Maybe the actual problem is in that collection?

[cidrbl0ck]

Update: So using buildah I modified the standard ansible-automation-platform23-ee-supported-rhel8 image to remove: Collections: kubernetes.core arista ibm

Python Packages: openshift-clients

Then installed: community.windows collection

Committing using buildah alleviated the task of building with ansible-builder, pushed it to my private automation hub and it's been used so far in roughly 40 deploys between 6 templates. All Windows, no issues.

Stinks that there's no methods of excluding in AB, also stinks that whatever was included in the BASE image wholly ignored by bindep.txt, and requirements.txt. This might not be the most elegant solution but it certainly worked until such time as the underlying openshift-clients issue is resolved in ee-supported-rhel8.

[fletort]

I have also this problem... Building an EE image from an ubi8 base image. I am adding the kubernetes.core collection so the assemble step failed as the rpm 'openshift-clients' is missing. But i don"t need openshift feature of the kubernetes.core collection... only the kubernetes feature....; It is a mess because only one module of this collection need this dependency..; no the other one.....

We should have a way to block some dependencies listed by introspect...

I will try to find a way to install the collection after the intrrospect step, maybe in the final build stage...

[cidrbl0ck]

Just an update for anyone fighting the same problem, since the issue here is 6 months old and there's no progress. (Admittedly this is a very small issue) Since I do not build EE's for distribution anywhere outside out internal Private Automation Hub I have continued to use Buildah with zero problems. I realize there's some Ansiblthonic-ness that I'm willfully excluding but there have been no drawbacks from this method so it will continue into the foreseeable future.

[kbreit]

I'm experiencing a need for excluding bindep packages. In my case a collection is trying to install python which doesn't exist and I want to exclude it. Outside of the suggested Buildah approach, I haven't figured out a way to make this work.

[cidrbl0ck]

You can’t just remove the collection with the python requirement?

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 12:53 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

I'm experiencing a need for excluding bindep packages. In my case a collection is trying to install python which doesn't exist and I want to exclude it. Outside of the suggested Buildah approach, I haven't figured out a way to make this work.

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961755444, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYAU6XN43M67UAPRLO3YVDJTDAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRG42TKNBUGQ. You are receiving this because you were mentioned.Message ID: @.***>

[kbreit-insight]

You can’t just remove the collection with the python requirement?

The collection is an important piece of the automation so I can't drop the collection.

[cidrbl0ck]

Just wanting to understand so I can help. If the collection requires Python.. but python isn’t available. Then I assume that does not break the entire collection but possibly parts that you don’t use?

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 1:30 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

You can’t just remove the collection with the python requirement?

The collection is an important piece of the automation so I can't drop the collection.

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961802966, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYHPRV5DXQ7L3T5AQNLYVDN3ZAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRHAYDEOJWGY. You are receiving this because you were mentioned.Message ID: @.***>

[cidrbl0ck]

Something else.. how does your EE not have python installed?

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 1:30 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

You can’t just remove the collection with the python requirement?

The collection is an important piece of the automation so I can't drop the collection.

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961802966, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYHPRV5DXQ7L3T5AQNLYVDN3ZAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRHAYDEOJWGY. You are receiving this because you were mentioned.Message ID: @.***>

[kbreit-insight]

Just wanting to understand so I can help. If the collection requires Python.. but python isn’t available. Then I assume that does not break the entire collection but possibly parts that you don’t use?

Python is installed on the EE. Specifically, I'm installing https://github.com/ansible/galaxy_collection which, in bindep, appears to try to install the package named python. The AAP RHEL8 minimal image, which I'm using, includes python3.9 out of the box.

I started a container using podman and here's the output if I try to install python.

bash-4.4# microdnf install python
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
Downloading metadata...
error: No package matches 'python'

[cidrbl0ck]

Ahhh ok interesting. I have a PAH but never needed or looked into interacting with it from a playbook/template.

So, am I understanding correctly that this Role does not detect that the EE has python installed? I’ve always used the supported RBI’s instead of the minimals. A microdnf update in a ee-minimal-rhel8 image just now resulted in 12-ish python39 related packages being updated. The bindep file lists python >=3.5, so to manually install or search (repoquery) you’d have to use python3* or python39 as the latest before any possible results come up.

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 1:48 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

Just wanting to understand so I can help. If the collection requires Python.. but python isn’t available. Then I assume that does not break the entire collection but possibly parts that you don’t use?

Python is installed on the EE. Specifically, I'm installing https://github.com/ansible/galaxy_collection which, in bindep, appears to try to install the package named python. The AAP RHEL8 minimal image, which I'm using, includes python3.9 out of the box.

I started a container using podman and here's the output if I try to install python.

bash-4.4# microdnf install python

Downloading metadata...

Downloading metadata...

Downloading metadata...

Downloading metadata...

Downloading metadata...

error: No package matches 'python'

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961825826, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYDQCSIASIHXPAIIEMTYVDQAVAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRHAZDKOBSGY. You are receiving this because you were mentioned.Message ID: @.***>

[cidrbl0ck]

You didn’t say but, I assume you are installing it via Ansible-galaxy yeah?

https://galaxy.ansible.com/ui/repo/published/sean_m_sullivan/tower_utilities/

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 1:48 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

Just wanting to understand so I can help. If the collection requires Python.. but python isn’t available. Then I assume that does not break the entire collection but possibly parts that you don’t use?

Python is installed on the EE. Specifically, I'm installing https://github.com/ansible/galaxy_collection which, in bindep, appears to try to install the package named python. The AAP RHEL8 minimal image, which I'm using, includes python3.9 out of the box.

I started a container using podman and here's the output if I try to install python.

bash-4.4# microdnf install python

Downloading metadata...

Downloading metadata...

Downloading metadata...

Downloading metadata...

Downloading metadata...

error: No package matches 'python'

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961825826, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYDQCSIASIHXPAIIEMTYVDQAVAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRHAZDKOBSGY. You are receiving this because you were mentioned.Message ID: @.***>

[kbreit-insight]

You didn’t say but, I assume you are installing it via Ansible-galaxy yeah?

Mostly correct. I'm installing it using ansible-galaxy but I'm actually installing it directly from GitHub because there's a bug in the collection that is fixed in git but hasn't been released to galaxy.ansible.com.

collections:
  - name: git+https://github.com/ansible/galaxy_collection
    type: git

[cidrbl0ck]

Gotcha. Im curious.. have you given the supported image a try.. just for curiosities sake?

There’s around 50 packages in the supported that’s not in the minimal… if it was me and I really wanted to use the minimal I’d try the other just to see.

But in any case.. other than perhaps the python syntax I mentioned earlier I don’t think I can help much more.

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 2:34 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

You didn’t say but, I assume you are installing it via Ansible-galaxy yeah?

Mostly correct. I'm installing it using ansible-galaxy but I'm actually installing it directly from GitHub because there's a bug in the collection that is fixed in git but hasn't been released to galaxy.ansible.com.

collections:

• name: git+https://github.com/ansible/galaxy_collection

  type: git

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961881720, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYCF7LFHGHHEBICIIHDYVDVMRAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRHA4DCNZSGA. You are receiving this because you were mentioned.Message ID: @.***>

[kbreit-insight]

Gotcha. Im curious.. have you given the supported image a try.. just for curiosities sake? There’s around 50 packages in the supported that’s not in the minimal… if it was me and I really wanted to use the minimal I’d try the other just to see. But in any case.. other than perhaps the python syntax I mentioned earlier I don’t think I can help much more.

The supported EE is not as well suited for my use case as image size is important for my situation. However, I gave it a try anyways and it failed with the same error as the python package doesn't exist for most/all RHEL environments.

My feeling is there are 3 options to fix it in descending order of quality. The Buildah options is in here somewhere as well.

1. Update the collection's bindep file to be more precise about the distributions that package should be installed for.
2. Add support for excluding packages in ansible-builder.
3. It seems the EE's have an /etc/dnf/dnf.conf file which should be able to take an excludes= option. However, I'm not seeing when it's created so more investigation is needed. This may be the fastest way to fix the problem though.

[cidrbl0ck]

And you are positive that the role/collection attempts to install python.. just that name exactly right? If so that sounds like a bug to me.. unless the collection was never meant for an RBI? Microdnf obviously doesn’t match on search/repoquery terms like full-fledged dnf does.

I’m not a huge Python guy but does removing/commenting out the python line in the bindep.txt file help at all?

From: Kevin Breit @.> Date: Friday, February 23, 2024 at 2:54 PM To: ansible/ansible-builder @.> Cc: Ben Hart @.>, Mention @.> Subject: Re: [ansible/ansible-builder] Correctly excluding a package? (Issue #472)

Gotcha. Im curious.. have you given the supported image a try.. just for curiosities sake? There’s around 50 packages in the supported that’s not in the minimal… if it was me and I really wanted to use the minimal I’d try the other just to see. But in any case.. other than perhaps the python syntax I mentioned earlier I don’t think I can help much more.

The supported EE is not as well suited for my use case as image size is important for my situation. However, I gave it a try anyways and it failed with the same error as the python package doesn't exist for most/all RHEL environments.

My feeling is there are 3 options to fix it in descending order of quality. The Buildah options is in here somewhere as well.

1. Update the collection's bindep file to be more precise about the distributions that package should be installed for.
2. Add support for excluding packages in ansible-builder.
3. It seems the EE's have an /etc/dnf/dnf.conf file which should be able to take an excludes= option. However, I'm not seeing when it's created so more investigation is needed. This may be the fastest way to fix the problem though.

— Reply to this email directly, view it on GitHubhttps://github.com/ansible/ansible-builder/issues/472#issuecomment-1961905363, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AST3YYG3RT7C6BEFZSJZFYTYVDXYBAVCNFSM6AAAAAAW4QEUGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRHEYDKMZWGM. You are receiving this because you were mentioned.Message ID: @.***>

[kbreit-insight]

And you are positive that the role/collection attempts to install python.. just that name exactly right? If so that sounds like a bug to me.. unless the collection was never meant for an RBI? Microdnf obviously doesn’t match on search/repoquery terms like full-fledged dnf does. I’m not a huge Python guy but does removing/commenting out the python line in the bindep.txt file help at all?

If I comment out the collection from requirements.yml the EE builds fine. I hope it is supposed to work with a UBI becaseu moving to the Red Hat supported automation_hub one will be tough unless they release a new version with the bug fix I need.

Regarding the bindep file, my next step on Monday is to create a fork, kill that file, and see if that works as an interm solution.