search

pulp / pulpcore

Pulp 3 pulpcore package https://pypi.org/project/pulpcore/
GNU General Public License v2.0
282 stars 111 forks source link

Remote artifactes are not created per remote when different remotes use same url #3240

Closed ipanova closed 1 week ago

ipanova commented 1 year ago

Version main and probably older

$ pulp rpm repository create --name a
{
  "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/09de57a2-f909-4cf5-bc1f-60ec40463afd/",
  "pulp_created": "2022-09-20T12:52:03.537255Z",
  "versions_href": "/pulp/api/v3/repositories/rpm/rpm/09de57a2-f909-4cf5-bc1f-60ec40463afd/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/09de57a2-f909-4cf5-bc1f-60ec40463afd/versions/0/",
  "name": "a",
  "description": null,
  "retain_repo_versions": null,
  "remote": null,
  "autopublish": false,
  "metadata_signing_service": null,
  "retain_package_versions": 0,
  "metadata_checksum_type": null,
  "package_checksum_type": null,
  "gpgcheck": 0,
  "repo_gpgcheck": 0,
  "sqlite_metadata": false
}

(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ pulp rpm repository create --name b
{
  "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/c2e726cd-564b-406a-bf32-838e22a39c06/",
  "pulp_created": "2022-09-20T12:52:06.158114Z",
  "versions_href": "/pulp/api/v3/repositories/rpm/rpm/c2e726cd-564b-406a-bf32-838e22a39c06/versions/",
  "pulp_labels": {},
  "latest_version_href": "/pulp/api/v3/repositories/rpm/rpm/c2e726cd-564b-406a-bf32-838e22a39c06/versions/0/",
  "name": "b",
  "description": null,
  "retain_repo_versions": null,
  "remote": null,
  "autopublish": false,
  "metadata_signing_service": null,
  "retain_package_versions": 0,
  "metadata_checksum_type": null,
  "package_checksum_type": null,
  "gpgcheck": 0,
  "repo_gpgcheck": 0,
  "sqlite_metadata": false
}

(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ pulp rpm remote create --name test-a --url https://fixtures.pulpproject.org/rpm-distribution-tree-changed-addon/
{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/022aefbc-7978-41d1-acfe-22c466d5b77a/",
  "pulp_created": "2022-09-20T12:52:19.130719Z",
  "name": "test-a",
  "url": "https://fixtures.pulpproject.org/rpm-distribution-tree-changed-addon/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "pulp_last_updated": "2022-09-20T12:52:19.130735Z",
  "download_concurrency": null,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "sles_auth_token": null
}

(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ pulp rpm repository sync --name a --remote /pulp/api/v3/remotes/rpm/rpm/022aefbc-7978-41d1-acfe-22c466d5b77a/
Started background task /pulp/api/v3/tasks/33368957-25af-4048-8d23-bdef1bb9cc00/
................................Done.
(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ pulp rpm remote create --name test-b --url https://fixtures.pulpproject.org/rpm-distribution-tree-changed-addon/
{
  "pulp_href": "/pulp/api/v3/remotes/rpm/rpm/c9d779cd-d290-49e1-a93b-a5b1646bdbb2/",
  "pulp_created": "2022-09-20T12:53:55.556938Z",
  "name": "test-b",
  "url": "https://fixtures.pulpproject.org/rpm-distribution-tree-changed-addon/",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "pulp_last_updated": "2022-09-20T12:53:55.556955Z",
  "download_concurrency": null,
  "max_retries": null,
  "policy": "immediate",
  "total_timeout": null,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": null,
  "headers": null,
  "rate_limit": null,
  "sles_auth_token": null
}

(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ django-admin shell_plus
# Shell Plus Model Imports
from pulpcore.app.models.access_policy import AccessPolicy, Group
from django.contrib.auth.models import Permission, User
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from pulp_ansible.app.models import AnsibleCollectionDeprecated, AnsibleDistribution, AnsibleRepository, Collection, CollectionImport, CollectionRemote, CollectionVersion, CollectionVersionSignature, DownloadLog, GitRemote, RoleRemote
from pulpcore.app.models.role import GroupRole, Role, UserRole
from pulp_container.app.models import Blob, BlobManifest, ContainerDistribution, ContainerNamespace, ContainerPushRepository, ContainerRemote, ContainerRepository, Manifest, ManifestListManifest, ManifestSignature, ManifestSigningService, Tag
from pulpcore.app.models.upload import Upload, UploadChunk
from pulp_file.app.models import FileAlternateContentSource, FileContent, FileDistribution, FilePublication, FileRemote, FileRepository
from pulp_rpm.app.models.acs import RpmAlternateContentSource
from pulp_rpm.app.models.advisory import UpdateCollection, UpdateCollectionPackage, UpdateRecord, UpdateReference
from pulp_rpm.app.models.comps import PackageCategory, PackageEnvironment, PackageGroup, PackageLangpacks
from pulp_rpm.app.models.custom_metadata import RepoMetadataFile
from pulp_rpm.app.models.distribution import Addon, Checksum, DistributionTree, Image, Variant
from pulp_rpm.app.models.modulemd import Modulemd, ModulemdDefaults, ModulemdObsolete
from pulp_rpm.app.models.package import Package
from pulp_rpm.app.models.repository import RpmDistribution, RpmPublication, RpmRemote, RpmRepository, UlnRemote
from pulpcore.app.models.acs import AlternateContentSource, AlternateContentSourcePath
from pulpcore.app.models.base import Label
from pulpcore.app.models.content import Artifact, AsciiArmoredDetachedSigningService, Content, ContentArtifact, PulpTemporaryFile, RemoteArtifact, SigningService
from pulpcore.app.models.exporter import Export, ExportedResource, Exporter, FilesystemExport, FilesystemExporter, PulpExport, PulpExporter
from pulpcore.app.models.importer import Import, Importer, PulpImport, PulpImporter, PulpImporterRepository
from pulpcore.app.models.progress import GroupProgressReport, ProgressReport
from pulpcore.app.models.publication import BaseDistribution, ContentGuard, ContentRedirectContentGuard, Distribution, Publication, PublishedArtifact, PublishedMetadata, RBACContentGuard
from pulpcore.app.models.repository import Remote, Repository, RepositoryContent, RepositoryVersion, RepositoryVersionContentDetails
from pulpcore.app.models.status import ContentAppStatus
from pulpcore.app.models.task import CreatedResource, Task, TaskGroup, TaskSchedule, Worker
from pulpcore.app.models.telemetry import SystemID
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When
from django.utils import timezone
from django.urls import reverse
from django.db.models import Exists, OuterRef, Subquery
Python 3.10.5 (main, Jun  9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: RemoteArtifact.objects.count()
Out[1]: 49

In [2]: Package.objects.first()
Out[2]: <Package: duck>

In [3]: duck = Package.objects.first()

In [4]: duck.contentartifact_set
Out[4]: <django.db.models.fields.related_descriptors.create_reverse_many_to_one_manager.<locals>.RelatedManager at 0x7ff228e98580>

In [5]: duck.contentartifact_set.all()
Out[5]: <QuerySet [<ContentArtifact: pk=230c81b2-a97c-40e2-8818-4ebb2c4c7b1d>]>

In [6]: duck.contentartifact_set.all()[0]
Out[6]: <ContentArtifact: pk=230c81b2-a97c-40e2-8818-4ebb2c4c7b1d>

In [7]: ca= duck.contentartifact_set.all()[0]

In [8]: ca.remoteartifact_set.all()
Out[8]: <RemoteArtifactQuerySet [<RemoteArtifact: pk=c5759950-fde1-488d-8fa3-c97d62f67f45>]>

In [9]: ra= ca.remoteartifact_set.all()[0]

In [10]: ra.remote
Out[10]: <Remote (pulp_type=rpm.rpm): test-a>

In [11]:                                                                                                                                                                                                                                      
Do you really want to exit ([y]/n)? y
(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ pulp rpm repository sync --name b --remote /pulp/api/v3/remotes/rpm/rpm/c9d779cd-d290-49e1-a93b-a5b1646bdbb2/
Started background task /pulp/api/v3/tasks/c45f8f3a-178e-4423-8eea-fa400a3b3170/
.......Done.
(pulp) [vagrant@pulp3-source-fedora36 pulp-2to3-migration]$ django-admin shell_plus
# Shell Plus Model Imports
from pulpcore.app.models.access_policy import AccessPolicy, Group
from django.contrib.auth.models import Permission, User
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from pulp_ansible.app.models import AnsibleCollectionDeprecated, AnsibleDistribution, AnsibleRepository, Collection, CollectionImport, CollectionRemote, CollectionVersion, CollectionVersionSignature, DownloadLog, GitRemote, RoleRemote
from pulpcore.app.models.role import GroupRole, Role, UserRole
from pulp_container.app.models import Blob, BlobManifest, ContainerDistribution, ContainerNamespace, ContainerPushRepository, ContainerRemote, ContainerRepository, Manifest, ManifestListManifest, ManifestSignature, ManifestSigningService, Tag
from pulpcore.app.models.upload import Upload, UploadChunk
from pulp_file.app.models import FileAlternateContentSource, FileContent, FileDistribution, FilePublication, FileRemote, FileRepository
from pulp_rpm.app.models.acs import RpmAlternateContentSource
from pulp_rpm.app.models.advisory import UpdateCollection, UpdateCollectionPackage, UpdateRecord, UpdateReference
from pulp_rpm.app.models.comps import PackageCategory, PackageEnvironment, PackageGroup, PackageLangpacks
from pulp_rpm.app.models.custom_metadata import RepoMetadataFile
from pulp_rpm.app.models.distribution import Addon, Checksum, DistributionTree, Image, Variant
from pulp_rpm.app.models.modulemd import Modulemd, ModulemdDefaults, ModulemdObsolete
from pulp_rpm.app.models.package import Package
from pulp_rpm.app.models.repository import RpmDistribution, RpmPublication, RpmRemote, RpmRepository, UlnRemote
from pulpcore.app.models.acs import AlternateContentSource, AlternateContentSourcePath
from pulpcore.app.models.base import Label
from pulpcore.app.models.content import Artifact, AsciiArmoredDetachedSigningService, Content, ContentArtifact, PulpTemporaryFile, RemoteArtifact, SigningService
from pulpcore.app.models.exporter import Export, ExportedResource, Exporter, FilesystemExport, FilesystemExporter, PulpExport, PulpExporter
from pulpcore.app.models.importer import Import, Importer, PulpImport, PulpImporter, PulpImporterRepository
from pulpcore.app.models.progress import GroupProgressReport, ProgressReport
from pulpcore.app.models.publication import BaseDistribution, ContentGuard, ContentRedirectContentGuard, Distribution, Publication, PublishedArtifact, PublishedMetadata, RBACContentGuard
from pulpcore.app.models.repository import Remote, Repository, RepositoryContent, RepositoryVersion, RepositoryVersionContentDetails
from pulpcore.app.models.status import ContentAppStatus
from pulpcore.app.models.task import CreatedResource, Task, TaskGroup, TaskSchedule, Worker
from pulpcore.app.models.telemetry import SystemID
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When
from django.utils import timezone
from django.urls import reverse
from django.db.models import Exists, OuterRef, Subquery
Python 3.10.5 (main, Jun  9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: RemoteArtifact.objects.count()
Out[1]: 55

In [2]: Package.objects.first()
Out[2]: <Package: duck>

In [3]: duck = Package.objects.first()

In [4]: duck.contentartifact_set.all()
Out[4]: <QuerySet [<ContentArtifact: pk=230c81b2-a97c-40e2-8818-4ebb2c4c7b1d>]>

In [5]: ca= duck.contentartifact_set.all()[0]

In [6]: ca.remoteartifact_set.all()
Out[6]: <RemoteArtifactQuerySet [<RemoteArtifact: pk=c5759950-fde1-488d-8fa3-c97d62f67f45>]>

In [7]: ra= ca.remoteartifact_set.all()[0]

In [8]: ra.remote
Out[8]: <Remote (pulp_type=rpm.rpm): test-a>

In [9]: ra.remote.id
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [9], in <cell line: 1>()
----> 1 ra.remote.id

AttributeError: 'Remote' object has no attribute 'id'

In [10]: ra.remote_id
Out[10]: UUID('022aefbc-7978-41d1-acfe-22c466d5b77a')

In [11]: RemoteArtifact.objects.filter(remote='022aefbc-7978-41d1-acfe-22c466d5b77a')
Out[11]: <RemoteArtifactQuerySet [<RemoteArtifact: pk=84476cac-c253-4a98-90d0-7848841eefb5>, <RemoteArtifact: pk=8a95ae21-3cde-4e49-a72e-5647396f0557>, <RemoteArtifact: pk=889d8ba7-6698-4c81-ae3a-03199e247b97>, <RemoteArtifact: pk=aaa75f7a-7a2b-45a4-8906-c07844481237>, <RemoteArtifact: pk=352158f3-a2f7-446b-8265-141d66fc78fe>, <RemoteArtifact: pk=098c445b-8fea-4e05-a2b5-9a6eb06927d6>, <RemoteArtifact: pk=fb0323c5-5b40-47d7-be0b-ae74713c3711>, <RemoteArtifact: pk=00f55f87-622d-492a-905c-f525656d2850>, <RemoteArtifact: pk=f4ccd58b-24b7-40f6-bdd3-628a88014ddd>, <RemoteArtifact: pk=7ca66eed-f9cf-4f5b-aab6-db2d31a97d13>, <RemoteArtifact: pk=516fe7cb-4d22-4aa0-901b-18316394cc07>, <RemoteArtifact: pk=d8fe7e54-92dc-4f09-928c-60ba1dcd5ab6>, <RemoteArtifact: pk=934aebcd-4f56-4c85-a4de-b839bb5fa968>, <RemoteArtifact: pk=e7b3b079-4473-458c-b4d4-64c23a903a1d>, <RemoteArtifact: pk=fe713a2f-4224-48bd-80a9-637639294bb2>, <RemoteArtifact: pk=76b6ec40-c7b2-4b47-9ccd-ccc91931878c>, <RemoteArtifact: pk=7b353459-35d7-4e81-892f-5434871e41ba>, <RemoteArtifact: pk=f379dd37-72cd-42dd-b14a-04f58c849fb3>, <RemoteArtifact: pk=1e853592-2ef3-43fd-a263-43d5a52d71fd>, <RemoteArtifact: pk=ee90a002-c9f7-4307-a343-35cef69d7179>, '...(remaining elements truncated)...']>

In [12]: RemoteArtifact.objects.filter(remote='c9d779cd-d290-49e1-a93b-a5b1646bdbb2')
Out[12]: <RemoteArtifactQuerySet [<RemoteArtifact: pk=21fee5c1-8e7d-4333-aefb-7acd1c8baa2c>, <RemoteArtifact: pk=ba2ea121-66b0-4ba7-9568-2d38719058a3>, <RemoteArtifact: pk=6cab62cc-a7d4-4503-b706-46c6b59f473e>, <RemoteArtifact: pk=0396bbda-4801-4627-b89e-fb73bbd0f626>, <RemoteArtifact: pk=d94cde5a-47e7-4c38-913c-1c4f970be8ca>, <RemoteArtifact: pk=39c8ff9c-23b5-42f8-9f91-e67d6e423b23>]>

In [13]: RemoteArtifact.objects.filter(remote='c9d779cd-d290-49e1-a93b-a5b1646bdbb2').count()
ipanova commented 1 year ago

it appears to be rpm only ( so far) issue. container seems to work properly

In [2]: Manifest.objects.first()
Out[2]: <Manifest: pk=0f00aa5a-f6a9-4ffe-a165-399b8fbb261d>

In [3]: m= Manifest.objects.first()

In [4]: m
Out[4]: <Manifest: pk=0f00aa5a-f6a9-4ffe-a165-399b8fbb261d>

In [5]: m.contentartifact_set
Out[5]: <django.db.models.fields.related_descriptors.create_reverse_many_to_one_manager.<locals>.RelatedManager at 0x7f2e9441bd00>

In [6]: m.contentartifact_set.all()
Out[6]: <QuerySet [<ContentArtifact: pk=16e513f6-a453-47ff-b735-528c5b7fc371>]>

In [7]: ca=m.contentartifact_set.all()[0]

In [8]: ca.remoteartifact_set
Out[8]: <django.db.models.fields.related_descriptors.create_reverse_many_to_one_manager.<locals>.RelatedManager at 0x7f2e9443be20>

In [9]: ca.remoteartifact_set.all()
Out[9]: <RemoteArtifactQuerySet [<RemoteArtifact: pk=f4c27061-4bd3-4c61-a614-9bd52719ceb7>, <RemoteArtifact: pk=e7162670-8d23-4e5d-a0c4-9f145193bdef>]>
ipanova commented 1 year ago

maybe this has some impact RemoteArtifactSaver(fix_mismatched_remote_artifacts=True) in the rpm pipeline

ipanova commented 1 year ago

The usual suspect says the codepath is in pulpcore RemoteArtifactSaver(fix_mismatched_remote_artifacts=True) so moving accordingly

lubosmj commented 1 week ago

This has not bring any issues so far and there is no further evidence of this problem occurring anymore.