Please add a switch to turn off avatar downloads

[dbk-rabel]

Is your feature request related to a problem? Please describe. We sync collections with PAH from ansible galaxy. In this progress the namespace avatars are downloaded. But they can be from any third party source. This is bad because it seems to be a security issue and also it is difficult because of firewall rules that have to be adjusted potentially.

Describe the solution you'd like For us it would be sufficient to have an option to disable avatar downloads.

And it would be great to also make this option accessable via PAH WebUI, but I think I would have to create an additional RFE there with Redhat, once this is implemented here.

Yours David

[mdellweg]

FWIW, I believe failing to download the avatar (by the power of firewall) should not impact the correctness and success of the sync. And for the security concerns, the validity of the avatar is checked by it's sha256.

Still one could add a tracker in the server of the avatar to gain intel on who is synching from a namespace.

[dbk-rabel]

Thanks for your answer.

I think you might be wrong though.

Here are some old logs from when we were first experiencing the problem. We put the collection jfrog.platform in the requirements.yml and started a collection sync in PAH, but did not have a proxy rule to allow access to media.jfrog.com

<30> 2024-01-18T17:08:33.152813+01:00 <hostname> pulpcore-worker[414996]: pulp [5b58bf5c37cd48bc81a8fe8fa00bd3bc]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 78ed5305-0272-4e0e-8aa4-afa951799ee9
<30> 2024-01-18T17:08:33.169612+01:00 <hostname> pulpcore-worker[414996]: pulp [5b58bf5c37cd48bc81a8fe8fa00bd3bc]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 78ed5305-0272-4e0e-8aa4-afa951799ee9
<30> 2024-01-18T17:08:41.248619+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: pulpcore.tasking.pulpcore_worker:INFO: Starting task aa26e234-44a9-42fb-8156-8b5470d5f7b8
<30> 2024-01-18T17:09:01.042546+01:00 <hostname> pulpcore-worker[415009]: Backing off download_wrapper(...) for 0.2s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])
<30> 2024-01-18T17:09:01.042546+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: backoff:INFO: Backing off download_wrapper(...) for 0.2s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])
<30> 2024-01-18T17:09:01.043035+01:00 <hostname> pulpcore-worker[415009]: Backing off download_wrapper(...) for 1.0s (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])

>>>> hundreds of lines more with the same content here <<<<

<30> 2024-01-18T17:09:36.952334+01:00 <hostname> pulpcore-worker[415009]: Giving up download_wrapper(...) after 4 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])
<30> 2024-01-18T17:09:36.952334+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: backoff:ERROR: Giving up download_wrapper(...) after 4 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])
<30> 2024-01-18T17:09:36.952334+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: pulp_ansible.app.tasks.collections:INFO: Failed to download namespace avatar: None - Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer], Skipping
<30> 2024-01-18T17:09:38.037557+01:00 <hostname> pulpcore-worker[415009]: Giving up download_wrapper(...) after 4 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])
<30> 2024-01-18T17:09:38.037557+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: backoff:ERROR: Giving up download_wrapper(...) after 4 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer])
<30> 2024-01-18T17:09:38.037990+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: pulp_ansible.app.tasks.collections:INFO: Failed to download namespace avatar: None - Cannot connect to host media.jfrog.com:443 ssl:default [Connection reset by peer], Skipping
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: pulpcore.tasking.pulpcore_worker:INFO: Task aa26e234-44a9-42fb-8156-8b5470d5f7b8 failed ('NoneType' object is not iterable)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]: pulp [2034f7d0143f4383bdd9df5df44397bc]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 458, in _perform_task
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    result = func(*args, **kwargs)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 191, in sync
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    repo_version = d_version.create()
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    loop.run_until_complete(pipeline)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    return future.result()
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 220, in create_pipeline
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    await asyncio.gather(*futures)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 41, in __call__
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    await self.run()
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/content_stages.py", line 198, in run
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    await sync_to_async(process_batch)()
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    ret = await asyncio.wait_for(future, timeout=None)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    return await fut
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    result = self.fn(*self.args, **self.kwargs)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    return func(*args, **kwargs)
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/content_stages.py", line 126, in process_batch
<30> 2024-01-18T17:09:38.350274+01:00 <hostname> pulpcore-worker[415009]:    for d_artifact in d_content.d_artifacts:

[mdellweg]

Can you confirm that your version runs with this DeclarativeFailsafeArtifact statement? https://github.com/pulp/pulp_ansible/blob/main/pulp_ansible/app/tasks/collections.py#L713

[dbk-rabel]

Seems like this was commit https://github.com/pulp/pulp_ansible/commit/f56e0974df20a53f778cf217fd4e7f64adca19c7 and therefore released with 0.21.3

It seems we are on version 0.17.5 . But the fix is also included there. (I checked the code on our system and also there is this commit: https://github.com/pulp/pulp_ansible/commit/6c6fefba18ca9c44d296a4784c87efc81a0e6f44 )

And additionally I just saw that the error message we receive wouldn't probably have been there before that commit.

[mdellweg]

Oh, the skipping seems to work. But there's another bug appearing: "'NoneType' object is not iterable" Can you confirm, this happens on the newest version too?

[dbk-rabel]

Ah ok.

"'NoneType' object is not iterable" should be fixed via https://github.com/pulp/pulp_ansible/pull/1813 according to Redhat support. But that has not made it in to a PAH release yet.

So you say that the namespace avatar was never the problem?

[mdellweg]

So you say that the namespace avatar was never the problem?

I'm saying the skipping of downloads works as advertised. But I don't yet understand the "real" cause of the stacktrace enough to say whether that is related.

[dbk-rabel]

Can I provide any more information to help with that?

[mdellweg]

If you want to do some debugging, It would be interesting to know which stage throws the error. Also can you spot a place, where d_artifacts is set to None?

[dbk-rabel]

Actually it seems that I am not able to reproduce the issue at the moment. :( Still got the old logs from January though.

[mdellweg]

Thanks for looking into this. If you get to see it again, we should have bugreport issue for it.

Let's keep this issue as a wishlist item. The original ask is valid as is (though rather low priority on our side).