relyt0925
commented
3 years ago I looked further in this use case and it looks like only the first tag is retried
Tylers-MacBook-Pro:armada-bootstrap tylerlisowski$ kubectl --kubeconfig /tmp/59555688-admin-kubeconfig get imagestreams -n openshift nginx -o yaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/display-name: Nginx HTTP server and a reverse proxy (nginx)
openshift.io/image.dockerRepositoryCheck: "2021-11-15T05:06:41Z"
samples.operator.openshift.io/version: 4.8.18
creationTimestamp: "2021-11-15T04:54:55Z"
generation: 3
labels:
samples.operator.openshift.io/managed: "true"
name: nginx
namespace: openshift
resourceVersion: "23068"
uid: 833d32c9-855e-4e84-a7ed-d51cc4b38a9d
spec:
lookupPolicy:
local: false
tags:
- annotations:
description: Build and serve static content via Nginx HTTP server and a reverse
proxy (nginx) on RHEL 8. For more information about using this builder image,
including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.14/README.md.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy 1.14 (RHEL
8)
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx
version: "1.14"
from:
kind: DockerImage
name: registry.redhat.io/rhel8/nginx-114:latest
generation: 3
importPolicy: {}
name: 1.14-el8
referencePolicy:
type: Local
- annotations:
description: Build and serve static content via Nginx HTTP server and a reverse
proxy (nginx) on RHEL 7. For more information about using this builder image,
including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.16/README.md.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy 1.16
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx,hidden
version: "1.16"
from:
kind: DockerImage
name: registry.redhat.io/rhscl/nginx-116-rhel7:latest
generation: 2
importPolicy: {}
name: "1.16"
referencePolicy:
type: Local
- annotations:
description: Build and serve static content via Nginx HTTP server and a reverse
proxy (nginx) on RHEL 7. For more information about using this builder image,
including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.16/README.md.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy 1.16 (RHEL
7)
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx
version: "1.16"
from:
kind: DockerImage
name: registry.redhat.io/rhscl/nginx-116-rhel7:latest
generation: 2
importPolicy: {}
name: 1.16-el7
referencePolicy:
type: Local
- annotations:
description: Build and serve static content via Nginx HTTP server and a reverse
proxy (nginx) on RHEL 8. For more information about using this builder image,
including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.16/README.md.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy 1.16 (RHEL
8)
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx
version: "1.16"
from:
kind: DockerImage
name: registry.redhat.io/rhel8/nginx-116:latest
generation: 2
importPolicy: {}
name: 1.16-el8
referencePolicy:
type: Local
- annotations:
description: Build and serve static content via Nginx HTTP server and a reverse
proxy (nginx) on RHEL 7. For more information about using this builder image,
including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.18/README.md.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy 1.18 (UBI 7)
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx
version: "1.18"
from:
kind: DockerImage
name: registry.redhat.io/ubi7/nginx-118:latest
generation: 2
importPolicy: {}
name: 1.18-ubi7
referencePolicy:
type: Local
- annotations:
description: Build and serve static content via Nginx HTTP server and a reverse
proxy (nginx) on RHEL 8. For more information about using this builder image,
including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.18/README.md.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy 1.18 (UBI 8)
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx
version: "1.18"
from:
kind: DockerImage
name: registry.redhat.io/ubi8/nginx-118:latest
generation: 2
importPolicy: {}
name: 1.18-ubi8
referencePolicy:
type: Local
- annotations:
description: |-
Build and serve static content via Nginx HTTP server and a reverse proxy (nginx) on RHEL. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/nginx-container/blob/master/1.18/README.md.
WARNING: By selecting this tag, your application will automatically update to use the latest version of Nginx available on OpenShift, including major version updates.
iconClass: icon-nginx
openshift.io/display-name: Nginx HTTP server and a reverse proxy (Latest)
openshift.io/provider-display-name: Red Hat, Inc.
sampleRepo: https://github.com/sclorg/nginx-ex.git
supports: nginx
tags: builder,nginx
from:
kind: ImageStreamTag
name: 1.18-ubi8
generation: 1
importPolicy: {}
name: latest
referencePolicy:
type: Local
status:
dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/nginx
tags:
- items:
- created: "2021-11-15T05:06:41Z"
dockerImageReference: registry.redhat.io/rhel8/nginx-114@sha256:c9e035d14cc8681307275eb62a2b2a054048121941ba70e68e1c6ac89dc875ed
generation: 3
image: sha256:c9e035d14cc8681307275eb62a2b2a054048121941ba70e68e1c6ac89dc875ed
tag: 1.14-el8
- conditions:
- generation: 2
lastTransitionTime: "2021-11-15T04:54:56Z"
message: 'Internal error occurred: registry.redhat.io/rhscl/nginx-116-rhel7:latest:
Get "https://registry.redhat.io/v2/": EOF'
reason: InternalError
status: "False"
type: ImportSuccess
items: null
tag: "1.16"
- conditions:
- generation: 2
lastTransitionTime: "2021-11-15T04:54:56Z"
message: 'Internal error occurred: registry.redhat.io/rhscl/nginx-116-rhel7:latest:
Get "https://registry.redhat.io/v2/": EOF'
reason: InternalError
status: "False"
type: ImportSuccess
items: null
tag: 1.16-el7
- conditions:
- generation: 2
lastTransitionTime: "2021-11-15T04:54:56Z"
message: 'Internal error occurred: registry.redhat.io/rhel8/nginx-116:latest:
Get "https://registry.redhat.io/v2/": EOF'
reason: InternalError
status: "False"
type: ImportSuccess
items: null
tag: 1.16-el8
- conditions:
- generation: 2
lastTransitionTime: "2021-11-15T04:54:56Z"
message: 'Internal error occurred: registry.redhat.io/ubi7/nginx-118:latest:
Get "https://registry.redhat.io/v2/": EOF'
reason: InternalError
status: "False"
type: ImportSuccess
items: null
tag: 1.18-ubi7
- conditions:
- generation: 2
lastTransitionTime: "2021-11-15T04:54:56Z"
message: 'Internal error occurred: registry.redhat.io/ubi8/nginx-118:latest:
Get "https://registry.redhat.io/v2/": EOF'
reason: InternalError
status: "False"
type: ImportSuccess
items: null
tag: 1.18-ubi8
Background: for a period of time since the openshift-apiserver now is wired to talk into the cluster for image imports image imports will fail. This is because konnectivity needs to startup and connect to the konnectivity server before image imports can start working.
What I have seen is image imports that hit an "internalError" only retry the first tag if there are multiple tags in an image stream. I have verified this on multiple image streams.
Trying to determine if this is a bug or expected? If expected hypershift might need to add logic to delete these and allow these to rereconcile after konnectivity has fully initalized
What I am seeing is these imports (anything that does not have an updated field is failed are never retried and if apps try and utilize them the new-app command fails since the image import is bad. This is expected for a period of time (when konnectivity is getting provisioned and cluster-dns is getting setup since konnectivity runs in the pod network). During that time: konnectivity-agent won't be able to connect back to the master. However, these image imports proceed and end up in a failed state.
Work around is to delete them and they will get reconciled and retried but wondering what a more graceful permanent solution is.