lubosmj
commented
3 weeks ago Was this working before? Did you upgrade to a newer version of pulpcore recently? Did you try to replicate the pull-through remotes and distributions?
Open PotentialIngenuity opened 3 weeks ago
lubosmj
commented
3 weeks ago Was this working before? Did you upgrade to a newer version of pulpcore recently? Did you try to replicate the pull-through remotes and distributions?
lubosmj
commented
3 weeks ago I wonder how did you end up in this state. I have checked out the main branch from pulpcore and pulp_container and the listing seems to be working fine on a fresh installation:
(venv3) [lmjachky@lmjachky-thinkpadt14gen4 pulpcore]$ http :5001/pulp/api/v3/distributions/container/pull-through/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Correlation-ID: 5dab5a3052574a22a1128ff2372f2127
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 30 Oct 2024 09:20:35 GMT
Referrer-Policy: same-origin
Server: nginx/1.22.1
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
{
"count": 0,
"next": null,
"previous": null,
"results": []
}
(venv3) [lmjachky@lmjachky-thinkpadt14gen4 pulpcore]$ http :5001/pulp/api/v3/remotes/container/pull-through/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Correlation-ID: 5ae9a1b76e954d44a4258898799181f1
Cross-Origin-Opener-Policy: same-origin
Date: Wed, 30 Oct 2024 09:20:40 GMT
Referrer-Policy: same-origin
Server: nginx/1.22.1
Vary: Accept
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
{
"count": 0,
"next": null,
"previous": null,
"results": []
}
PotentialIngenuity
commented
3 weeks ago I can see we had a successful replication a week ago. I just found it in a broken state yesterday because I went to update pulp itself. It was at 3.63.0 and went to 3.66.0. Both versions were in this broken state. I do see the pull through assets on the replica.
I was able to list the container repos in the catalog. The list is the same between the source and the replica.
{
"repositories":[
"docker-cache/balabit/syslog-ng"
"docker-cache/library/alpine"
"docker-cache/library/balabit/syslog-ng"
"docker-cache/library/busybox"
"docker-cache/library/centos"
"docker-cache/library/golang"
"docker-cache/library/hello-world"
"docker-cache/library/nginx"
"docker-cache/library/postgres"
"docker-cache/library/postgres-backup-local"
"docker-cache/library/prodrigestivill/postgres-backup-local"
"docker-cache/library/pulp/pulp-minimal"
"docker-cache/library/pulp/pulp-web"
"docker-cache/library/rabbitmq"
"docker-cache/library/redis"
"docker-cache/library/rockylinux"
"docker-cache/postgres"
"docker-cache/prodrigestivill/postgres-backup-local"
"docker-cache/pulp/pulp-minimal"
"docker-cache/pulp/pulp-web"
"docker-cache/redis"
"docker-cache/zabbix/rabbitmq"
"docker-cache/zabbix/zabbix-agent"
"quay-cache/ansible/awx"
"quay-cache/ansible/awx-ee"
"quay-cache/ansible/awx-manager"
"quay-cache/ansible/awx-operator"
"quay-cache/ansible/awx-task"
"quay-cache/library/ansible/awx"
"quay-cache/library/awx"
]
}Interestingly I can pull any container from both source/replica that is not a pulp container.
❯ skopeo list-tags docker://registry.cobaltiron.com/docker-cache/library/pulp/pulp-web
{
"Repository": "registry.cobaltiron.com/docker-cache/library/pulp/pulp-web",
"Tags": []
}
❯ skopeo list-tags docker://registry.cobaltiron.com/docker-cache/pulp/pulp-web
FATA[0002] Error listing repository tags: fetching tags list: name unknown: Repository not found.I was able to get the pulp containers working by deleting the remotes/distributions and then pulling them again.
docker pull registry.company.com/docker-cache/pulp/pulp-web:3.66.0
3.66.0: Pulling from docker-cache/pulp/pulp-web
b2cc5146c9c7: Already exists
782ee99b4ec9: Already exists
28c6947581d3: Already exists
f622eee2e866: Already exists
02ada972a7c9: Already exists
0f0aa4efd0d2: Already exists
346b0e450128: Already exists
fe083a31111c: Already exists
d7e5fb241397: Pull complete
Digest: sha256:e4c880250611e2b867775e1c2c9bb20ca6ed06a033fc25a50f4ccff1340011c9
Status: Downloaded newer image for registry.company.com/docker-cache/pulp/pulp-web:3.66.0
registry.company.com/docker-cache/pulp/pulp-web:3.66.0I want to try to delete the remote/distribution and recreate them for the docker-cache pull-through but I cant list it to get the href.
PotentialIngenuity
commented
3 weeks ago Side question: why do some repositories have library in the name and some do not. Which one should be used to pull?
PotentialIngenuity
commented
15 hours ago Any ideas on how to fix the KeyError: 'container.remote' error? This is still happening on 3.68.0
Version core: 3.66.0 container: 2.21.1
Describe the bug The pull through caches cannot be listed. Some of them can be pulled from but some of them cannot be.
I can see that the pull through still exists but trying to create it again.
This container pulled just fine
This one does not pull
The 500 error
The same 500 error happens during a replicate tasks also.
To Reproduce n/a
Expected behavior The container plugin should function as designed.
Additional context I did not make any changes leading up to this error. I saw it on core 3.63.0 and updated to 3.66.0 to see if it went away but it did not.