Open Gabrielandre02 opened 10 months ago
Hi,
We are facing the same issue. In addition to the reported details, the gitlab registry cleanup policy when enabled will generate plenty of logs in both sides due to images not being deleted.
We got several instances, but the one we are facing this issue in is using registry 2.8.x instead of 2.7.x.
It seems related to several things :
Such an issue has been reported before
/v2/<repository>/manifests/<tag>
in 16.6.2 and /v2/<repository>/manifests/<digest without sha256 prefix>
making the deletion failIs there any documented way to use gitlab registry with sameersbn image ?
Hi,
unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry.
However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image.
Did anyone try this?
Hi @sachilles,
unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry.
This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?
However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image.
Did anyone try this?
I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\
I'm also experiencing errors when deleting images from the container registry. I noticed the problem after upgrading to 16.6. I suppose it is connected to Gitlab forking the Docker distribution registry. In my case I'm currently using registry:2.7.1
from Docker Distribution to host container registry images. @ymazzer @sachilles are you using the same version?
I actually stumbled on https://gitlab.com/gitlab-org/container-registry/-/issues/958#note_1471217687 that consider my setup (using registry:2.7.1
) as a third party registry that will not be supported in the future.
In that thread is suggested to switch to the gitlab-container-registry
version as declared in the Omnibus package. Apparently for Gitlab 16.6.2 the gitlab-container-registry
version is v3.86.1-gitlab
.
I tried the switch
/etc/docker/registry/config.yml
(apparently without an initial configuration file it won't start) registry:
- image: registry:2.7.1
+ image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.86.1-gitlab"
volumes:
+ - ./registry:/etc/docker/registry
- /data/gitlab/registry:/registry
- ./certs:/certs
I experienced an issue with file system permission during deletion, but I haven't figured out the correct user to switch to. I indeed temporary solved it by allowing write to everyone.
Hi @ymazzer,
This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?
I have the same problem. It doesn't matter if I try to delete container images via the GitLab web interface or via the cleanup policy. The result is the same. (I'm using the "official" self-hosted docker registry in version 2.8.1.)
I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\
Okay, I guess the proposed way is to migrate the content of the "official" self-hosted docker registry (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459).
Dear @avvertix,
I found somewhere (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459) that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry, as several changes were included after the container registry was forked by the GitLab maintainers. (My first attempt to replace the Docker registry with the GitLab container registry failed).
However, please make sure to backup the entire Docker registry data directories before the necessary migration.
Once I (or someone else) is successful, we will update the corresponding configuration files and documentation.
@avvertix, This solution solved the problem and related to deleting the images. Tested on version 16.7.0. however I had to clean all the images within the registry.
Thanks
https://gitlab.com/gitlab-org/gitlab/-/issues/423459 that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry
Dear @sachilles I also saw the link you shared, but my interpretation is that the issue refers to the migration from filesystem metadata (or object storage) to the database storage for the images metadata. As far as I understood migrating to the database metadata is in beta and not yet fully complete that's why I didn't try that way.
I saw a workaround following the [feedback issue]() on the use of external registries.
I totally agree that my approach is just a workaround, but probably could make easier the migration path until all official migrations tools are available.
however I had to clean all the images within the registry.
@Gabrielandre02 could you tell me more about why you had to clean all the images, as in the end I was able to use the same storage directory by ensuring that the user within the docker image had access to the filesystem. (btw seems that in my case the user on the host os didn't have write permission on a parent folder in the storage, after I ensured that all the folders/files are writeable by the current user everything worked without changing nothing)
Hi @sachilles @avvertix,
I just tried out the simple following procedure:
registry:2.8
, I replaced the image by the latest gitlab-registry registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
1000:1000
and then started the registry and tried building an OCI image.# docker-compose.yml
# [...]
registry:
#image: registry:2.8
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
restart: unless-stopped
expose:
- "5000"
ports:
- "127.0.0.1:5000:5000"
volumes:
- ./registry-config.yml:/etc/docker/registry/config.yml # this has changed as described before by @avvertix
- /opt/gitlab/disk/data/shared/registry:/var/lib/registry/docker/registry
- /etc/ssl/private/registry.my-registry.com:/certs
environment:
REGISTRY_LOG_LEVEL: info
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
REGISTRY_AUTH_TOKEN_REALM: https://my-gitlab.com/jwt/auth
REGISTRY_AUTH_TOKEN_SERVICE: container_registry
REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/cert.crt
REGISTRY_STORAGE_DELETE_ENABLED: "true"
networks:
- web
# [...]
Hope this will help.
PS: I obviously made a backup before doing the operation.
Will there be an update of the docs and an “official” way for migrating to the new docker image?
@etlam Yes, the project-related docs will be updated possible.
Whenever I try to delete an image or a tag, I'm getting this error but the logs in the registry aren't showing anything, nor are those in Gitlab. This installation was via a Linux package, I made the backup and left it in a new configuration with Docker Swarm, using Traefik as a reverse proxy
LOGS GITLAB:
LOGS REGISTRY:
application_json.log