search

harvester / tests

Harvester test cases
Apache License 2.0
10 stars 31 forks source link

[TEST] Build the fully airgapped on bare metal machines for release upgrade testing #967

Open TachunLin opened 10 months ago

TachunLin commented 10 months ago

What's the test to develop? Please describe

The epic issue was created to track the progress of building a fully airgapped environment for the upgrade testing in each Harvester release candidate.

Since the current fully airgapped environment was built upon ipxe-example virtual machines inside another powerful virtual machine. This would usually cause some unexpected upgrade failure due to performance or resource bottleneck.

Scope

  1. Provide a fully airgapped Harvester cluster on bare metal machines
  2. Provide a fully airgapped Rancher instance and private docker registry
  3. Provide an artifact sever have internet connection act as the file, dns and ntp server.
  4. Automatically import Harvester into Rancher
  5. Provision the downstream RKE2 cluster

Prerequisite

Any prerequisite environment and pre-condition required for this test. Provide test case dependency here if any.

The fully airgapped environment requires the following components:

  1. Four bare metal machines for Harvester cluster
  2. One bare machine to host Rancher and private registry
  3. One VM for the http and ntp server
  4. At lease two external vlan network
  5. Each node machine should provide more than 16 core CPU, 32GB memory, 500GB nvme disks can meet Harvester recommended hardware requirement

Test case reference

Roles

Describe the items of the test development (DoD, definition of done) you'd like

Stage 1 Design discussion

Stage 2 Build Out Baseline Provision Harvester Airgap Pipeline

Stage 3 Convert Vagrant Logic To Terraform VMs (per service) For Harvester

Stage 4 Build Out All Airgap Integrations Jenkins Pipeline Utilities

Stage 5 Implement All Airgap Integrations Jenkins Pipeline on Staging

Stage 6 Implement additional pipeline parameters to baseline Harvester Airgap Pipeline

Stage 7 Move Both Pipelines To Prod

Stage 8 Implement New Pipeline To Run Subsection Of Tests Against Airgap

TachunLin commented 10 months ago

These are the initial concept, need further discussion and update by time.

Pipeline design concept

  1. Share the seeder machine resource with the existing upgrade test pipeline
  2. Utilize the current upgrade test pipeline to prepare a new pipeline for Harvester cluster provision only
  3. Require a separate pipeline, we can specify which version of Rancher we would like to provision
  4. If the Rancher instance was built on on VM, we can use the sshuttle or port forwarding to expose the 192.168.2.0 subnet to other host machine
  5. Use a separate pipeline to prepare and cached different versions of the offline Rancher image file
  6. Combine each pipeline with dependencies to perform certain task, e.g prepare the entire airgapped environment

Pipeline request

  1. Provision the Harvester cluster on bare machines
  2. Provision the airgapped Rancher
  3. Prepare the private docker registry for the specific Rancher offline image
  4. Build up the artifact server provide HTTP, DNS and NTP service
  5. The all-in-one pipeline to prepare the fully airgapped test environment ready
TachunLin commented 10 months ago

Just come up the initial idea of the fully airgapped infrastructure diagram for further discussion

image

irishgordo commented 10 months ago

Just come up the initial idea of the fully airgapped infrastructure diagram for further discussion

image

I think in general this looks pretty good :smile: :+1:

I would only mention that there would be much more benefit to separating out hp-176 two run two vagrant vms - one to serve the registry and one to serve rancher initially instead of at a later point as the provisioning w/ Rancher-&-Docker-Registry has many flaws currently in ipxe-examples: Screenshot from 2023-11-02 10-26-59

TachunLin commented 9 months ago

The Rancher instance, docker registry, DNS, name server implementation would be implemented in https://github.com/harvester/tests/issues/942

TachunLin commented 9 months ago

Another idea is we can consider moving the Artifact server role from the external VM to inside the hp-176 seeder machine. This may decrease the effort to handle the network connectivity and could be better utilize the airgapped network created by the Open vSwitch.

image

irishgordo commented 7 months ago

deployment drawio

irishgordo commented 3 months ago

There is a slight blocker at: https://github.com/harvester/harvester/issues/5301

Means we will need to bake in additional logic in the pipeline to compensate for that bug.

irishgordo commented 1 month ago

We're currently encountering something that we will need to redesign logic for. We're hitting:

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: -1: Map expressions can only contain up to 125 entries @ line -1, column -1.
1 error

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1107)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:635)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:581)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
    at hudson.model.ResourceController.execute(ResourceController.java:101)
    at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE

Seemingly related to something within Jenkins / Groovy :

Investigating....

irishgordo commented 1 month ago

Even pivoting, now we are hitting a limitation of the script string... Investigating


2024-07-23 00:05:09.979+0000 [id=26]    SEVERE  hudson.util.BootFailure#publish: Failed to initialize Jenkins
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
script: 280: String too long. The given string is 93362 Unicode code units long, but only a maximum of 65535 is allowed.
 @ line 280, column 20.
               script('''
                      ^

1 error

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1107)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:677)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:689)
    at groovy.lang.GroovyShell$parse.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
    at javaposse.jobdsl.dsl.AbstractDslScriptLoader.parseScript(AbstractDslScriptLoader.groovy:134)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:177)
    at javaposse.jobdsl.dsl.AbstractDslScriptLoader.runScriptEngine(AbstractDslScriptLoader.groovy:101)
Caused: javaposse.jobdsl.dsl.DslException: startup failed:
script: 280: String too long. The given string is 93362 Unicode code units long, but only a maximum of 65535 is allowed.
 @ line 280, column 20.
               script('''
irishgordo commented 1 month ago

Was able to reduce it, but still it is too big:

Caused: javaposse.jobdsl.dsl.DslException: startup failed:
script: 280: String too long. The given string is 71859 Unicode code units long, but only a maximum of 65535 is allowed.
 @ line 280, column 20.
               script('''
                      ^

1 error

We'll need to pivot to something else w/ jobdls plugin...

irishgordo commented 1 month ago

Based on some more investigation, I'm not entirely sure all integrations can be within a single pipeline job... Still investigating...

But in: https://github.com/jenkinsci/job-dsl-plugin/blob/e6d655dd5b2874f56af8bf4b99a4d622b752bb98/job-dsl-plugin/src/main/java/javaposse/jobdsl/plugin/JenkinsJobManagement.java#L258-L287

Where the JobDSL plugin, possibly is calling: readFileFromWorkspace() in:

pipelineJob('example') {
    definition {
        cps {
            script(readFileFromWorkspace('project-a-workflow.groovy'))
            sandbox()
        }
    }
}

That it still ultimately in that, where we read a file... it seems to do from the jobdsl repo:

return filePath.readToString();

Ultimately, reading the file into a String, so we're back in the same place we would be even if we defined it as:

script(
'''
script in here
'''
)

^ because that also just yields a "String". So we can't rip it out into a file to escape the:

 String too long. The given string is 71859 Unicode code units long, but only a maximum of 65535 is allowed.

Though... I'm not entirely sure about this. My initial thinking is that we would need to break this up into "multiple" pipeline jobs ... Example:

So that then scales, all our jobs from 1 (that provisions all integrations)... to needing to be "multiple"... one per airgap integration... possibly to just avoid this Groovy limitation of the string being too big :sweat_smile: ... Again, not entirely sure though....

irishgordo commented 1 month ago

With: https://github.com/irishgordo/harvester-baremetal-ansible/commit/b80e3dde3a36281bb7e861c5fb2c0956d66473f4 Was able to reduce it down so that the "String too large" error disappeared.

But still, the underlying error is now present as it's just seeing the script function as being too large in general...

Investigating the new error of:

Started by user [admin](http://172.19.98.192:8083/user/admin)
Running as [admin](http://172.19.98.192:8083/user/admin)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
General error during class generation: Method too large: WorkflowScript.___cps___1 ()Lcom/cloudbees/groovy/cps/impl/CpsFunction;

groovyjarjarasm.asm.MethodTooLargeException: Method too large: WorkflowScript.___cps___1 ()Lcom/cloudbees/groovy/cps/impl/CpsFunction;
    at groovyjarjarasm.asm.MethodWriter.computeMethodInfoSize(MethodWriter.java:2087)
    at groovyjarjarasm.asm.ClassWriter.toByteArray(ClassWriter.java:447)
    at org.codehaus.groovy.control.CompilationUnit$17.call(CompilationUnit.java:850)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:635)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:581)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
    at hudson.model.ResourceController.execute(ResourceController.java:101)
    at hudson.model.Executor.run(Executor.java:442)

1 error

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1107)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:635)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:581)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
    at hudson.model.ResourceController.execute(ResourceController.java:101)
    at hudson.model.Executor.run(Executor.java:442)
Finished: FAILURE
irishgordo commented 1 month ago

Trying:

        JAVA_OPTS: "-Dorg.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=true -Djenkins.install.runSetupWizard=false -Djenkins.install.SetupWizard.adminInitialApiToken=\"{{ lookup('password', '/dev/null length=20 chars=ascii_letters') }}\" -Dhudson.model.DirectoryBrowserSupport.CSP=\"\""

Specifically:

-Dorg.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=true

As suggested:

Has led to the same result... Pivoting to other solutions...

irishgordo commented 1 month ago

Timeboxing... Was trying variations of:

    definition {
        cpsScm {
            scm {
                git {
                    remote {
                        github('${harvester_baremetal_ansible_repo}', '${harvester_baremetal_ansible_branch}')
                        credentials('github-credential')
                    }
                }
                scriptPath("jenkins/harvester_airgap_integrations_pipeline.groovy")
            }
        }
    }

It's really not working... getting the params.* to come across the wire and be interpolated into things is just simply not working with the combinations of:

With the idea, we'd give a specific script path like:

                scriptPath("jenkins/harvester_airgap_integrations_pipeline.groovy")

to split apart script / pipeline But getting the branch & repo dynamically... through interpolation of the params (stringParam) type...

May pivot back to cpsScm -> scm -> git & scriptPath...

The Jenkins JobDSL Plugin Docs + Jenkins Docs don't seem to have "dynamic" examples...

irishgordo commented 1 month ago

Re-investigating the environment. That would be the easiest... thinking that with some adjustments than: https://github.com/harvester/tests/issues/967#issuecomment-2243737049 ( that error )

irishgordo commented 1 month ago

It's difficult to overcome the "environment variable" limit... there are probably still some more ways around it...

Pivoted instead to:

With:

Methods, that get around the "method too large" error, since we pull then the logic of the multiple parallel running stages out into two separate methods -> one to build out the local.tfvars for the respective service, since we can't leverage the default environment TF_VAR_* that Terraform gives us because Jenkins/Groovy is placing a strange limitation on the map size of the environment variables with the JobDSL plugin.

If we could, we'd avoid an entire parallel stage that's needed to build out the local.tfvars.

That leverages the second bullet point from:

irishgordo commented 1 month ago

What ended up working for interpolation and also syncing with the needed style of the local.tfvars for each service is using the $/ string...

def string = $/
string-goes-here
${params.interpolation}

other things like newline\n
/$

To seemingly help

irishgordo commented 1 month ago

Currently, testing pipeline on staging... Will iterate to fix any outstanding bugs as everything is now becoming glued together...

irishgordo commented 1 month ago

So, the temporary loop to do a few iterations when we shift the VM NIC/NAD and run a separate playbook for airgap seems to help buffer:

But the second iteration we're still seeing:

│ <172.19.121.147> (0, b'', b"OpenSSH_9.7p1, OpenSSL 3.3.1 4 Jun 2024\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 22: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug2: resolve_canonicalize: hostname 172.19.121.147 is address\r\ndebug1: auto-mux: Trying existing master at '/var/jenkins_home/.ansible/cp/fa3d4b2f87'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 17473\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")

│ fatal: [dns-server-argp-vm]: FAILED! => {

│     "msg": "Timeout (12s) waiting for privilege escalation prompt: "

│ }

│ 

│ PLAY RECAP *********************************************************************

│ dns-server-argp-vm         : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

│ 

│ 

│ 

│   with ansible_playbook.dns-vm-ansible-playbook,

│   on main.tf line 178, in resource "ansible_playbook" "dns-vm-ansible-playbook":

│  178: resource "ansible_playbook" "dns-vm-ansible-playbook" {

│ 

│ ansible-playbook

Implementing an arbitrary timeout of sleep, prior to the next iteration so the VM can make the DHCP request outbound and get the IPv4 assigned regardless of network, will need to happen.

irishgordo commented 1 month ago

Something is happening and the /etc/rancher/k3s/registries.yaml isn't getting the injected variable funneled in from Jenkins:

root@k3s-server-argp-vm:/home/ubuntu# cat /etc/rancher/k3s/registries.yaml 
mirrors:
  docker.io:
    endpoint:
      - "https://airgap-docker-registry..sslip.io:5000"
  registry.suse.com:
    endpoint:
      - "https://airgap-docker-registry..sslip.io:5000"
configs:
  "airgap-docker-registry..sslip.io:5000":
    tls:
      insecure_skip_verify: true
  "https://airgap-docker-registry..sslip.io:5000":
    tls:
      insecure_skip_verify: true
  "":
    tls:
      insecure_skip_verify: true
root@k3s-server-argp-vm:/home/ubuntu#

investigating...

irishgordo commented 1 month ago

This was an issue:

\"stderr\": \"Error: open /home/ubuntu/hauler-jetstack-cert-manager-images.yaml: no such file or directory\\nUsage:\\n  hauler store sync [flags]\\n\\nFlags:\\n  -f, --files strings             Path(s) to local content files (Manifests). i.e. '--files ./rke2-files.yml\\n  -h, --help                      help for sync\\n  -k, --key string                (Optional) Path to the key for signature verification\\n  -p, --platform string           (Optional) Specific platform to save. i.e. linux/amd64. Defaults to all if flag is omitted.\\n  -c, --product-registry string   (Optional) Specific Product Registry to use. Defaults to RGS Carbide Registry (rgcrprod.azurecr.us).\\n      --products strings          Used for RGS Carbide customers to supply a product and version and Hauler will retrieve the images. i.e. '--product rancher=v2.7.6'\\n  -r, --registry string           (Optional) Default pull registry for image refs that are not specifying a registry name.\\n\\nGlobal Flags:\\n      --cache string       (deprecated flag and currently not used)\\n  -l, --log-level string    (default \\\"info\\\")\\n  -s, --store string       Location to create store at (default \\\"store\\\")\",\n    \"stderr_lines\": [\n        \"Error: open /home/ubuntu/hauler-jetstack-cert-manager-images.yaml: no such file or directory\",\n        \"Usage:\",\n        \"  hauler store sync [flags]\",\n        \"\",\n        \"Flags:\",\n        \"  -f, --files strings             Path(s) to local content files (Manifests). i.e. '--files ./rke2-files.yml\",\n        \"  -h, --help                      help for sync\",\n        \"  -k, --key string                (Optional) Path to the key for signature verification\",\n        \"  -p, --platform string           (Optional) Specific platform to save. i.e. linux/amd64. Defaults to all if flag is omitted.\",\n        \"  -c, --product-registry string   (Optional) Specific Product Registry to use. Defaults to RGS Carbide Registry (rgcrprod.azurecr.us).\",\n        \"      --products strings          Used for RGS Carbide customers to supply a product and version and Hauler will retrieve the images. i.e. '--product rancher=v2.7.6'\",\n        \"  -r, --registry string           (Optional) Default pull registry for image refs that are not specifying a registry name.\",\n        \"\",\n        \"Global Flags:\",\n        \"      --cache string       (deprecated flag and currently not used)\",\n        \"  -l, --log-level string    (default \\\"info\\\")\",\n        \"  -s, --store string       Location to create store at (default \\\"store\\\")\"\n    ],\n    \"stdout\": \"\\u001b[90m2024-07-26 22:52:17\\u001b[0m \\u001b[1m\\u001b[31mERR\\u001b[0m\\u001b[0m open /home/ubuntu/hauler-jetstack-cert-manager-images.yaml: no such file or directory\",\n    \"stdout_lines\": [\n        \"\\u001b[90m2024-07-26 22:52:17\\u001b[0m \\u001b[1m\\u001b[31mERR\\u001b[0m\\u001b[0m open /home/ubuntu/hauler-jetstack-cert-manager-images.yaml: no such file or directory\"\n    ]\n}\n\nTASK [seed-hauler : Print when errors] *****************************************\ntask path: /var/jenkins_home/workspace/harvester-airgap-integrations/terraform/airgap-integrations/hauler/ansible/roles/seed-hauler/tasks/main.yml:52\nok: [hauler-server-argp-vm] =\u003e {\n    \"msg\": \"I caught an error in configuring vm further\"\n}\n\nTASK [seed-hauler : Always do this] ********************************************\ntask path: /var/jenkins_home/workspace/harvester-airgap-integrations/

Now fixed from Sunday's update. Yielding:

╭─mike at suse-workstation-team-harvester in ~/Projects/seeder/cmd/seeder on cli✘✘✘
╰─± curl -k https://172.19.121.240:5000/v2/library/nginx/tags/list | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    43  100    43    0     0    346      0 --:--:-- --:--:-- --:--:--   349
{
  "name": "library/nginx",
  "tags": [
    "latest"
  ]
}
╭─mike at suse-workstation-team-harvester in ~/Projects/seeder/cmd/seeder on cli✘✘✘
╰─± ./hauler store add image quay.io/jetstack/cert-manager-webhook:v1.13.1 -p linux/amd64
╭─mike at suse-workstation-team-harvester in ~/Projects/seeder/cmd/seeder on cli✘✘✘
╰─± curl -k https://airgap-docker-registry.172.19.121.240.sslip.io:5000/v2/jetstack/cert-manager-cainjector/tags/list | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    63  100    63    0     0    195      0 --:--:-- --:--:-- --:--:--   195
{
  "name": "jetstack/cert-manager-cainjector",
  "tags": [
    "v1.13.1"
  ]
}

So cert-manager & nginx are present.

Additionally all:

rescue:

"rescue" blocks in Ansible, will re-trigger an ansible.builtin.fail with a message giving context at a glance, allowing the pipelines to not "fail silently"