Unable to pull postgres image from local/private registry

[CWollinger]

ISSUE TYPE

• Bug Report

SUMMARY

I am not able to pull the postgres image from a local/private registry. I tried it with different docker-tokens and tags.

ENVIRONMENT

• AWX version: 19.2.2
• Operator version: 0.12.0
• Kubernetes version: v1.20.8
• AWX install method: kubernetes

STEPS TO REPRODUCE

1. Push the postgres:12 image to a private registry
2. Create the secret for the docker token in the namespace
3. Create the awx spec
4. Apply the new awx

EXPECTED RESULTS

No problems with the image push like for redis, awx, ee.

ACTUAL RESULTS

Pod fails with the following error:

Error: ImagePullBackOff
Pulling image "my-internal.reg.example.com/docker/postgres:12"
Failed to pull image "my-internal.reg.example.com/docker/postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "my-internal.reg.example.com/docker/postgres:12": failed to resolve reference "my-internal.reg.example.com/docker/postgres:12": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

If I start other deployments on the node and namespace with the same image there no problems. The other images are pulled without problem and the awx deployment is running.

ADDITIONAL INFORMATION

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  image_pull_secret: docker-token
  postgres_image: my-internal.reg.example.com/docker/new_postgres
  image: my-internal.reg.example.com/docker/awx
  ee_images:
    - name: awx-ee
      image: my-internal.reg.example.com/docker/new_ee:0.5.0
  redis_image: my-internal.reg.example.com/docker/awx_redis
  redis_image_version: 5.0.9

[urskog84]

I experiance the same problem!

[RosarioVinoth]

I hit the same problem as well.

[tchellomello]

We should be covered by https://github.com/ansible/awx-operator/blob/devel/roles/installer/templates/postgres.yaml.j2#L36-L39

{% if image_pull_secret %}
      imagePullSecrets:
        - name: {{ image_pull_secret }}
{% endif %}

Having said this, could you show us how you created your secret?

$ kubectl get secret docker-token -n awx -o yaml     #remember to obfuscate the password

Also, what happens if you do the following:

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
  namespace: awx
spec:
  containers:
  - name: private-reg-container
    image: my-internal.reg.example.com/docker/new_postgres:12
  imagePullSecrets:
  - name:  docker-token

Please note the jinja2 template expected the '{{ postgres_image }}:{{ postgres_image_version }}' where postgres_image_version by default is 12 and NOT devel. On your private registry, do you have the image as devel or 12?

[CWollinger]

Secret created from the dockerconfigjson:

apiVersion: v1
data:
  .dockerconfigjson: base64
kind: Secret
metadata:
  creationTimestamp: "2021-07-08T09:42:40Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:.dockerconfigjson: {}
      f:type: {}
    manager: kubectl-create
    operation: Update
    time: "2021-07-08T09:42:40Z"
  name: docker-token
  namespace: awx
  resourceVersion: "36332729"
  selfLink: /api/v1/namespaces/awx/secrets/docker-token
  uid: 969d6c33-18d1-4737-af9f-xx
type: kubernetes.io/dockerconfigjson

The creation of a pod or deployment with this image and secret is working without any problem. The image tag in the registry is 12

[ntlartey]

I'm currently hitting this same issue. Is there explanation why this doesn't work when you explicitly specify a private registry, image and pull secret and defaults, I don't know how to a quay.io image?

[lals1]

Experiencing the same problem with pulling the image from private registry.