Gerrit docker volume permissions issue

[mgrzechocinski]

Hi.

I'm struggling with strange Gerrit issues after upgrading from 2.14.x (do not remember which exactly) to 2.15.2. I started a discussion on Gerrit mailing list here but I was led to the point where there's a strange behaviour of Gerrit running in a container made out of this image.

When I run this image through docker-compose and log into the container by docker exec -u gerrit2 -it gerrit-container I'm able to browse my gerrit review site. What looks strange for me is the permissions in var/gerrit/review_site/git/training.git/refs/changes:

bash-4.4$ ls -al
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 07:37 88
drwxr-xr-x    4 gerrit2  root           136 Jul 27 08:46 90
drwxr-xr-x    4 gerrit2  root           136 Jul 27 08:58 93
drwxr-xr-x    4 root  root              136 Jul 27 09:24 94
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 09:33 95
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 11:46 97
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 13:39 98
drwxr-xr-x    4 gerrit2  root           136 Jul 27 14:35 99

How is it possible that some folders are owned by gerrit2, some by root and same with group it. This state of my volume causes many strange errors when publishing changes or trying to merge them.

Frankly, I have no idea why this happens. I run Docker on Mac. My docker-compose looks like this:

version: '3'

services:
  nginx:
    container_name: nginx-container
    build:
      context: ./nginx
      dockerfile: Dockerfile
    volumes:
      - ~/Docker/infra_mobiledev/nginx/conf/:/etc/nginx/conf.d/
      - ~/Docker/infra_mobiledev/nginx/fancyindex_theme_1/:/usr/share/nginx/fancyindex_theme:ro
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - gerrit
  gerrit:
    container_name: gerrit-container
    image: openfrontier/gerrit:latest
    volumes:
      - ~/Docker/infra_mobiledev/gerrit_home:/var/gerrit/review_site
    environment:
    - MIGRATE_TO_NOTEDB_OFFLINE=1
    - AUTH_TYPE=HTTP
    - WEBURL=https://localhost/gerrit/
    - AUTH_EMAIL_FORMAT={0}@example.org
    - HTTPD_LISTENURL=proxy-https://gerrit:8080/gerrit/
    - GERRIT_INIT_ARGS=--install-plugin=download-commands
    - GERRIT_START_ACTION=daemon
    ports:
      - "29418:29418"

[thinkernel]

Hi @mgrzechocinski This sounds like a serious problem. This happens when you upgrade from 2.14.x to 2.15.2, doesn't it. When you restart the container everything is OK because this line works. However we should still find out how the owner is changed in the first place. And further more why only parts of them is changed.

drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 07:37 88
drwxr-xr-x    4 gerrit2  root           136 Jul 27 08:46 90 <- this is rare but has no side effect
drwxr-xr-x    4 gerrit2  root           136 Jul 27 08:58 93
drwxr-xr-x    4 root  root               136 Jul 27 09:24 94 <-why only this change is owned by root?
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 09:33 95
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 11:46 97
drwxr-xr-x    3 gerrit2  gerrit2        102 Jul 27 13:39 98
drwxr-xr-x    4 gerrit2  root           136 Jul 27 14:35 99

If this is changed by upgrade process I think all of them should be changed to "root root". Do you still keep the repository before the upgrade? It would be very helpful if you can tell what these changes' owner look like before the upgrade.

[mgrzechocinski]

Hi. Thanks for the response. I'll try to restore my backup from 2.14.2, run it and see how files permissions look like. Need a 1-2 days to get back to you.

Meanwhile, do you know whether something has changed recently regarding permissions? AFAIK, all files in review_sile should have gerrit2:gerrit2 access, right?

[mgrzechocinski]

What I can tell so far is that I can easily reproduce it on my review_site when publishing two changes, one depending on the other. More about this here. I've never faced this issue when publishing single change, merging it, then publihsing another change, rebased on the merged one.

[thinkernel]

I see. Let me try reproducing this issue in my container. If it only happens on two depending changes, it might be a bug of Gerrit itself.

[thinkernel]

Hi @mgrzechocinski I don't know if it's a good news or a bad news for you. However, I managed to create a 2.14.9 container and use importer plugin to import 3 projects from another living gerrit container. After that, I upgrade this container to 2.15.2 twice, with MIGRATE_TO_NOTEDB_OFFLINE=1 and without MIGRATE_TO_NOTEDB_OFFLINE. I checked the permission in the git directory both time after the upgrading process. All directories and files' permission are gerrit2:gerrit2. I'm going to push 2 depending changes. I'll tell you the result later. ps. I figure out that it's not a good idea to keep MIGRATE_TO_NOTEDB_OFFLINE in an environmental variable since we don't want to migrate to notedb again after we reboot the container. I'll try to doing a reboot to see what happens.

[thinkernel]

I create 2 depending changes and push it to gerrit together. After that I review it first one then do the submit. Everything is fine. Then I do the same thing on the second one. Still nothing wrong. However I observe the changes under refs/changes. After the pushing, there are 2 directories and 2 empty sub-directories under the changes which are 12/1212 and 13/1213. Yes, just two empty directories are created. Then I do the review. After that there's a meta file created under one directory. After I submit the change the meta file is disappeared. You may wonder where's the file named 1 which used to be under the 1212 and 1213 directories going. I noticed there is a file named packed-refs created under the projectname.git directory. Just cat this file you can figure out why the file named as 1 is unnecessary anymore. I think this is how the notedb works. BTW, I use http to do the push, not ssh. And beside notedb I use postgres as the backend storage. I think postgres is still necessary until 2.16 is released. My container is running in k8s. So I don't use docker-compose as yours but I think it's not a matter.

The only thing that might be matter is I use ceph rbd as the volume provider. Yours is a local directory. Since the permission in a local directory is quite easy to be changed by another process running in your computer. That might be a hind to figure out what happened.

And you've never mentioned if there are any shell scripts under /docker-entrypoint-init.d which will be running during the startup.

[thinkernel]

Hi @mgrzechocinski I read your compose file. You should check the permission of ~/Docker/infra_mobiledev/gerrit_home on your host. And my don't recommend mount directory under ~ directory since *nix system have different way to deal with permission policy on personal home directory. I recommend using named volumes which is created by docker volume command or something like that in docker-compose file. Named volumes will be created under /var/lib/docker/volumes and docker can manage the permission on it correctly.

[mgrzechocinski]

Hi.

Thanks for the detailed response. I will definitely try to move my volumes out of the user's dir and check folder permissions. I have an impression that sometimes it works, sometimes it doesn't - just like there would be any other (host) process which breaks those permissions in the background.

I put my files as a local volume in user's dir, because it's way easier to access them by my backup system. I forgot to mention that I'm actually using Docker for Mac and AFAIR it's not so easy to access named volumes through host filesystem (to make a backup) in Docker for Mac.

[thinkernel]

Hi @mgrzechocinski I see. I recommend you use another docker container to backup the named volumes. Just mount the named volume you want to backup and zip everything to another volume which is a local directory mounted in the same container. Docker's documents have some advices about how to backup volumes.

[mgrzechocinski]

This story ended up with creating completely new instance of Gerrit 2.15.x, with PostgreSQL in separated container. Never had this issue anymore, even after upgrading to 2.16.x. Therefore, I'm going to close this issue.