Wrong package download after on-demand sync

[pedro-psb]

Originally reported here. I'm bringing it here for visibility in the RemoteArtifact bucket of issues.

Version

Confirmed to happen with (at least).

• pulpcore==3.21, pulp_rpm==3.18
• pulpcore==3.39, pulp_rpm==3.23

Describe the bug

When:

• Repos A and B syncs (on-demand) the same Content from two independent Remotes
• In Remote B, the Content is replaced (e.g in rpm, same Nevra different checksum) and synced again (on-demand).
• A request is made to the Content for Distribution associated with Repo A

Then:

• Pulp raises a digest validation error and retries to download from the same (wrong) remote
• Client can't get any content

To Reproduce

This is a flaky problem. Most likely it depends on sorting RemoteArtifact via uuid4, which is random. I'll write a Pulp test and share it here to make it easier to re-run.

1. Serve a signed package in two different "external" (non-pulp) repos. E.g, download sample, run createrepo and make sure it's reachable from the Pulp instance.
2. For each of the two repos:
   • Create a Repository (autopublish=True), Remote (policy=on_demand and corresponding url) and Distribution
   • sync (policy=mirror_content_only)
3. Change one of the external repos to serve the unsigned version of the package. E.g, delete the signed, put this one, run createrepo and sync the corresponding Pulp repo again.
4. Request the package from the distribution whose package hasnt changed (should contain signed) and see the error.

Expected behavior

The right package should be delivered to the client, as its present and reachable in Remote A.

Additional context

• On previous versions (core/rpm 3.16/3.17), the client would effectively get the wrong package, but that's not the case in supported versions anymore.
• Overview of what is happening in Pulp:

[lubosmj]

We can adjust the order_by_acs() function or add a new sorting: 1. ACS wins if available, 2. remote associated with repo/distribution, 3. the rest is random (randomizing all the remotes).

The discussion around copying content will come.