Upgrade GitLab to v17.4.1

[SilinPavel]

Background It would be great to have possibility to deploy CloudPipeline with fresh GitLab version (17.4.1 as of now)

[SilinPavel]

GitLab upgrade from v15.5.4 to 17.4.1

Initial setup:

• postgres:12.4
• gitlab:15.5.4

Based on :

• Upgrade path tool: https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/?current=15.5.0&edition=ce
• PostgreSQL matrix: https://docs.gitlab.com/ee/administration/package_information/postgresql_versions.html
• Installation reqs: https://docs.gitlab.com/ee/install/requirements.html#postgresql

Update process will look as follow:

Uprade steps

0. Backup current stage
1. Restore backup to the temp GitLab deployment
2. -> postgres:14.11
3. -> Migrate PAC tokens if needed
4. -> gitlab:15.11.13
5. -> gitlab:16.3.8
6. -> gitlab:16.7.9
7. -> gitlab:16.11.10
8. -> gitlab:17.3.3
9. -> gitlab:17.4.1
10. Create backup and migrate it to the kubernetes GitLab deployment

0. Backup current stage

1. Create backup:

   
   kubectl exec -it <gitlab gitlab-ctl stop

gitlab-backup create

kubectl exec -it <gitlab gitlab-ctl start

gitlab_settings_bkp_file="/var/opt/gitlab/backups/settings-dump-$(date +%Y%m%d).tgz" tar -czf $gitlab_settings_bkp_file /etc/gitlab

2. Save backup from /var/opt/gitlab/backups/ somewhere save and accessable

The following steps will restore created backup into temp gitlab deployment, then will update this deployment up to the latest version.
Finally, we will create another backup for the final version and as a last step - restore this backup to the kubernetes gitlab deployment. 

### 1. Restore backup to the temp GitLab deployment

Before upgrade let's prepare environment. First of all, create working directories:

mkdir $GITLAB_HOME/logs mkdir $GITLAB_HOME/config mkdir $GITLAB_HOME/data

Create gitlab.rb  file into config  directory with the following content:

gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'utf8' gitlab_rails['db_host'] = 'localhost' gitlab_rails['db_port'] = gitlab_rails['db_username'] = 'gitlab' gitlab_rails['db_password'] = 'gitlab' postgresql['enable'] = false

Let's prepare database:

docker run --name postgres12 -e POSTGRES_PASSWORD=postgres -d -p :5432 postgres:12.4 docker exec postgres9 psql -U postgres -c "create user gitlab createdb;" docker exec postgres9 psql -U postgres -c "alter user gitlab with superuser;" docker exec postgres9 psql -U postgres -c "alter user gitlab with password 'gitlab';" docker exec postgres9 psql -U postgres -c "create database gitlabhq_production owner gitlab;"

Now we are ready to run gitlab. This gitlab will be empty and connected to database refined below.

The version of gitlab shall be exactly the same as gitlab version from backup dumb. docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:12.5.4-ce.0

Once gitlab is ready and healthy:
 - stop gitlab (`docker stop gitlab`) 
 - copy `etc/gitlab` from cp-git-settings backup file to `$GITLAB_HOME/config` (except for `gitlab.rb` file)
 - place backup dumb into `$GITLAB_HOME/data/backups`.  
 - Start gitlab again: `docker restart gitlab`

Stop processes that may have connection with database and restore backup:

docker exec gitlab gitlab-ctl stop puma docker exec gitlab gitlab-ctl stop sidekiq

Verify

docker exec gitlab gitlab-ctl status

Restore

docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=

Answer yes for the first question:

Before restoring the database we recommend removing all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed.

Do you want to continue (yes/no)? yes

Answer no for the second:

This will rebuild an authorized_keys file. You will lose any data stored in authorized_keys file. Do you want to continue (yes/no)? no

Restart gitlab container and wait for healthy state:

docker restart gitlab

Check state

docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true

Now the gitlab copy is ready for upgrade.

### 2. Upgrade to postgres:14.11

Let's prepare database:

docker run --name postgres14 -e POSTGRES_PASSWORD=postgres -d -p :5432 postgres:14.11 docker exec postgres9 psql -U postgres -c "create user gitlab createdb;" docker exec postgres9 psql -U postgres -c "alter user gitlab with superuser;" docker exec postgres9 psql -U postgres -c "alter user gitlab with password 'gitlab';" docker exec postgres9 psql -U postgres -c "create database gitlabhq_production owner gitlab;"

Once PostgreSQL container prepared make a database backup:

Stop processes that are connected to DB docker exec gitlab gitlab-ctl stop puma docker exec gitlab gitlab-ctl stop sidekiq

Verify

docker exec gitlab gitlab-ctl status

Update `$GITLAB_HOME/config/gitlab.rb` file: postgresql['version'] = 14

Create database backup:

docker exec gitlab gitlab-ctl reconfigure docker exec gitlab gitlab-ctl stop

Create database backup and make a note of the backup file name

docker exec gitlab gitlab-rake gitlab:backup:create SKIP=repositories,uploads,builds,artifacts,lfs,pages,registry

Update `$GITLAB_HOME/config/gitlab.rb` file with a new DB configurations (to connect to the new db instance)

Restore backup:
> NOTE: <backup timestamp> is not a file name. This is a part before _gitlab_backup.tar . For example, if we have a backup file with name 1666196708_2022_10_19_12.0.12_gitlab_backup.tar the <backup timestamp> in this case will be 1666196708_2022_10_19_12.0.12 and the command will be:

gitlab-rake gitlab:backup:restore BACKUP=/var/opt/gitlab/backups/1666196708_2022_10_19_12.0.12 docker exec gitlab gitlab-ctl reconfigure

Restore database backup:

docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=

Answer yes for the first question:

Before restoring the database we recommend removing all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed.

Do you want to continue (yes/no)? yes

Answer no for the second:

This will rebuild an authorized_keys file. You will lose any data stored in authorized_keys file. Do you want to continue (yes/no)? no

docker exec gitlab gitlab-ctl start

wait for gitlab api up and check everything is ok: repositories, commit availability, etc

### 3. Migrate PAC tokens:
> Previous deployment db can have problematic PAC tokens, which will lead to the fail of the update of GitLab.
To fix this, connect to the GitLab db and run the following SQL update query: 

update personal_access_tokens set scopes = REPLACE(scopes, ' !ruby/array:Hashie::Array', '')

### 4. Upgrade to gitlab:15.11.13

### How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:15.11.13-ce.0

> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

### 5. Upgrade to gitlab:16.3.8

### How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:16.3.8-ce.0

> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

### 6. Upgrade to gitlab:16.7.9

### How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:16.7.9-ce.0

> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

### 7. Upgrade to gitlab:16.11.10

### How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:16.11.10-ce.0

> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

### 8. Upgrade to gitlab:17.3.3

### How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:17.3.3-ce.0

> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

### 9. Upgrade to gitlab:17.4.1

### How to upgrade gitlab instalce

docker stop gitlab && docker rm gitlab

docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:17.4.1-ce.0

> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished

### 10. Create backup and migrate it to the kubernetes GitLab deployment

#### Create backup
1. Create backup:

gitlab-backup create

gitlab_settings_bkp_file="/var/opt/gitlab/backups/settings-dump-$(date +%Y%m%d).tgz" tar -czf $gitlab_settings_bkp_file /etc/gitlab

2. Save backup from `/var/opt/gitlab/backups/` (host path: `$GITLAB_HOME/data/backups/`) somewhere save and accessable

#### Deploy new cp-gitlab-db pod
1. Create a new folder for `gitlab db` on the system share or local folder on a mater node which hosts all deployment dirs, f.e:

mkdir /opt/gitlab-17-postgresql/data
```

2. Using templates deploy/contents/k8s/cp-gitlab-db/cp-gitlab-db-dpl.yaml and deploy/contents/k8s/cp-gitlab-db/cp-gitlab-db-svc.yaml create fresh gitlab-db deployment
   • Change labels, selectors and names to be identical (f.e. cp-gitlab-db -> cp-gitlab-db-17)
   • Change volumes/gitlab-pgdata from /opt/gitlab-postgresql/data to /opt/gitlab-17-postgresql/data
   • Apply this configs with kubectl apply -f <file>

Create empty gitlab db:

Inside the gitlab-db container execute:

psql -U postgres -c "create user gitlab createdb;"
psql -U postgres -c "alter user gitlab with superuser;"
psql -U postgres -c "alter user gitlab with password 'gitlab';"
psql -U postgres -c "create database gitlabhq_production owner gitlab;"

Deploy new cp-gitlab pod

1. Create a new folder for gitlab on the system share or local folder on a mater node which hosts all deployment dirs, f.e:

   mkdir -p /opt/gitlab-17/logs
   mkdir -p /opt/gitlab-17/config
   mkdir -p /opt/gitlab-17/data
   mkdir -p /opt/gitlab-17/pki
   cp -r /opt/gitlab/pki/* /opt/gitlab-17/pki/

2. Using templates deploy/contents/k8s/cp-git/cp-git-dpl.yaml create fresh gitlab deployment
   • Change labels, selectors and names to be identical (f.e. cp-git -> cp-git-17)
   • Change volumes/git-logs from /opt/gitlab/logsto/opt/gitlab-17/logs. The same for volumesvolumes/git-config,volumes/git-data`volumes/git-pki
   • Override gitlab-db connection with containers/env section:

     
     env:

   • name: GITLAB_DATABASE_HOST value: f.e.: cp-gitlab-db-17.default.svc.cluster.local
   • name: GITLAB_DATABASE_PORT value: "6543"
   • name: GITLAB_ROOT_PASSWORD value:

   • Apply this configs with kubectl apply -f <file>

After appliyng:

Copy prepared backup

cp <backup location>/<backup>_gitlab_backup.tar /opt/gitlab/data/backups/
chmod a+r /opt/gitlab-17/data/backups/<backup>_gitlab_backup.tar

Restore backup

kubectl exec -it $GITLAB_POD gitlab-ctl stop puma
kubectl exec -it $GITLAB_POD gitlab-ctl stop sidekiq
kubectl exec -it $GITLAB_POD gitlab-rake gitlab:backup:restore BACKUP=$GITLAB_BACKUP
--- yes
--- no

Copy configuration files from settings-dump backup (except for gitlab.rb) "cp /etc/* /opt/gitlab/config/"

Reconfigure gitlab

kubectl exec -it $GITLAB_POD gitlab-ctl reconfigure

Restart git pod and check status
kubectl exec -it $GITLAB_POD gitlab-rake gitlab:check SANITIZE=true

Things to consider:

• Starting from GitLab 16, all tokens will be < 365 days lifetime, which means