Cannot communicate with Sonatype Nexus docker repository, fails with 401 with correct credentials

[Soraph]

Version The problem has been found with the following scenario:

Installed via Automation Platform 2.4

python39-pulp-container.noarch     2.15.5-1.el8ap
python39-pulpcore.noarch           3.28.32-1.el8ap

Installed via Automation Platform 2.5

python3.11-pulp-container.noarch   2.19.3-1.el9ap
python3.11-pulpcore.noarch         3.49.19-1.el9ap

Describe the bug When attempting to communicate with a registry of type Sonatype Nexus the authentication step does not happen correctly.

Same credentials used with podman allow proper login and operations.

To Reproduce Steps to reproduce the behavior:

Testing with Automation Hub 2.4

1. Create a new registry with the Sonatype Nexus URL and credentials
2. Define a new Execution Environment definition (image)
3. Attempt to synchronize it
4. 401 error

   pulpcore-worker[70580]: pulp [fa3fba1ec0eb4238b421812448c70e22]: pulpcore.tasking.tasks:INFO: Starting task 0192fb6b-5231-70e9-a1ad-9be63c8f897c
   pulpcore-worker[70580]: pulp [fa3fba1ec0eb4238b421812448c70e22]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=test remote=test
   pulpcore-worker[70580]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('http://nexus.domain/v2/'))
   pulpcore-worker[70580]: pulp [fa3fba1ec0eb4238b421812448c70e22]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('http://nexus.domain/v2/'))
   pulpcore-worker[70580]: pulp [fa3fba1ec0eb4238b421812448c70e22]: pulpcore.tasking.tasks:INFO: Task 0192fb6b-5231-70e9-a1ad-9be63c8f897c failed (Connection closed)
   pulpcore-worker[70580]: pulp [fa3fba1ec0eb4238b421812448c70e22]: pulpcore.tasking.tasks:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 66, in _execute_task
   pulpcore-worker[70580]:    result = func(*args, **kwargs)
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/synchronize.py", line 41, in synchronize
   pulpcore-worker[70580]:    return dv.create()
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
   pulpcore-worker[70580]:    loop.run_until_complete(pipeline)
   pulpcore-worker[70580]:  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
   pulpcore-worker[70580]:    return future.result()
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 220, in create_pipeline
   pulpcore-worker[70580]:    await asyncio.gather(*futures)
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 41, in __call__
   pulpcore-worker[70580]:    await self.run()
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 138, in run
   pulpcore-worker[70580]:    tag_list = await self.get_paginated_tag_list(tag_list_url, repo_name)
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/tasks/sync_stages.py", line 337, in get_paginated_tag_list
   pulpcore-worker[70580]:    await list_downloader.run(extra_data={"repo_name": repo_name, "headers": {}})
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 269, in run
   pulpcore-worker[70580]:    return await download_wrapper()
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
   pulpcore-worker[70580]:    ret = await target(*args, **kwargs)
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 254, in download_wrapper
   pulpcore-worker[70580]:    return await self._run(extra_data=extra_data)
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py", line 97, in _run
   pulpcore-worker[70580]:    to_return = await self._handle_response(response)
   pulpcore-worker[70580]:  File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 203, in _handle_response
   pulpcore-worker[70580]:    chunk = await response.content.read(1048576)  # 1 megabyte
   pulpcore-worker[70580]:  File "/usr/lib64/python3.9/site-packages/aiohttp/streams.py", line 347, in read
   pulpcore-worker[70580]:    raise self._exception

Expected behavior Authentication is happening correctly

Additional context As previously discussed with Lubos, the error seems to be related to the file /usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py where we check the following code:

elif "Basic" in response_auth_header:

While it should be something on the lines of

elif "basic" in response_auth_header.lower():

This should be backported to versions 2.15.x and 2.19.x if possible as it is affecting Automation Platform 2.4 and 2.5