Open SilinPavel opened 4 weeks ago
Initial setup:
Based on :
Update process will look as follow:
kubectl exec -it <gitlab gitlab-ctl stop
gitlab-backup create
kubectl exec -it <gitlab gitlab-ctl start
gitlab_settings_bkp_file="/var/opt/gitlab/backups/settings-dump-$(date +%Y%m%d).tgz" tar -czf $gitlab_settings_bkp_file /etc/gitlab
2. Save backup from /var/opt/gitlab/backups/ somewhere save and accessable
The following steps will restore created backup into temp gitlab deployment, then will update this deployment up to the latest version.
Finally, we will create another backup for the final version and as a last step - restore this backup to the kubernetes gitlab deployment.
### 1. Restore backup to the temp GitLab deployment
Before upgrade let's prepare environment. First of all, create working directories:
mkdir $GITLAB_HOME/logs mkdir $GITLAB_HOME/config mkdir $GITLAB_HOME/data
Create gitlab.rb file into config directory with the following content:
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'utf8'
gitlab_rails['db_host'] = 'localhost'
gitlab_rails['db_port'] =
Let's prepare database:
docker run --name postgres12 -e POSTGRES_PASSWORD=postgres -d -p
Now we are ready to run gitlab. This gitlab will be empty and connected to database refined below.
The version of gitlab shall be exactly the same as gitlab version from backup dumb. docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:12.5.4-ce.0
Once gitlab is ready and healthy:
- stop gitlab (`docker stop gitlab`)
- copy `etc/gitlab` from cp-git-settings backup file to `$GITLAB_HOME/config` (except for `gitlab.rb` file)
- place backup dumb into `$GITLAB_HOME/data/backups`.
- Start gitlab again: `docker restart gitlab`
Stop processes that may have connection with database and restore backup:
docker exec gitlab gitlab-ctl stop puma docker exec gitlab gitlab-ctl stop sidekiq
docker exec gitlab gitlab-ctl status
docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=
Answer yes for the first question:
Before restoring the database we recommend removing all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed.
Do you want to continue (yes/no)? yes
Answer no for the second:
This will rebuild an authorized_keys file. You will lose any data stored in authorized_keys file. Do you want to continue (yes/no)? no
Restart gitlab container and wait for healthy state:
docker restart gitlab
docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true
Now the gitlab copy is ready for upgrade.
### 2. Upgrade to postgres:14.11
Let's prepare database:
docker run --name postgres14 -e POSTGRES_PASSWORD=postgres -d -p
Once PostgreSQL container prepared make a database backup:
Stop processes that are connected to DB docker exec gitlab gitlab-ctl stop puma docker exec gitlab gitlab-ctl stop sidekiq
docker exec gitlab gitlab-ctl status
Update `$GITLAB_HOME/config/gitlab.rb` file: postgresql['version'] = 14
Create database backup:
docker exec gitlab gitlab-ctl reconfigure docker exec gitlab gitlab-ctl stop
docker exec gitlab gitlab-rake gitlab:backup:create SKIP=repositories,uploads,builds,artifacts,lfs,pages,registry
Update `$GITLAB_HOME/config/gitlab.rb` file with a new DB configurations (to connect to the new db instance)
Restore backup:
> NOTE: <backup timestamp> is not a file name. This is a part before _gitlab_backup.tar . For example, if we have a backup file with name 1666196708_2022_10_19_12.0.12_gitlab_backup.tar the <backup timestamp> in this case will be 1666196708_2022_10_19_12.0.12 and the command will be:
gitlab-rake gitlab:backup:restore BACKUP=/var/opt/gitlab/backups/1666196708_2022_10_19_12.0.12 docker exec gitlab gitlab-ctl reconfigure
docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=
Answer yes for the first question:
Before restoring the database we recommend removing all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed.
Do you want to continue (yes/no)? yes
Answer no for the second:
This will rebuild an authorized_keys file. You will lose any data stored in authorized_keys file. Do you want to continue (yes/no)? no
docker exec gitlab gitlab-ctl start
wait for gitlab api up and check everything is ok: repositories, commit availability, etc
### 3. Migrate PAC tokens:
> Previous deployment db can have problematic PAC tokens, which will lead to the fail of the update of GitLab.
To fix this, connect to the GitLab db and run the following SQL update query:
update personal_access_tokens set scopes = REPLACE(scopes, ' !ruby/array:Hashie::Array', '')
### 4. Upgrade to gitlab:15.11.13
### How to upgrade gitlab instalce
docker stop gitlab && docker rm gitlab
docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:15.11.13-ce.0
> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished
### 5. Upgrade to gitlab:16.3.8
### How to upgrade gitlab instalce
docker stop gitlab && docker rm gitlab
docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:16.3.8-ce.0
> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished
### 6. Upgrade to gitlab:16.7.9
### How to upgrade gitlab instalce
docker stop gitlab && docker rm gitlab
docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:16.7.9-ce.0
> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished
### 7. Upgrade to gitlab:16.11.10
### How to upgrade gitlab instalce
docker stop gitlab && docker rm gitlab
docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:16.11.10-ce.0
> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished
### 8. Upgrade to gitlab:17.3.3
### How to upgrade gitlab instalce
docker stop gitlab && docker rm gitlab
docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:17.3.3-ce.0
> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished
### 9. Upgrade to gitlab:17.4.1
### How to upgrade gitlab instalce
docker stop gitlab && docker rm gitlab
docker run -d --network=host \ --hostname gitlab.example.com \ --name gitlab \ --volume $GITLAB_HOME/config:/etc/gitlab \ --volume $GITLAB_HOME/logs:/var/log/gitlab \ --volume $GITLAB_HOME/data:/var/opt/gitlab \ --shm-size 256m \ gitlab/gitlab-ce:17.4.1-ce.0
> Check and wait: Admin area → Background jobs: all background_migrations jobs shall be finished
### 10. Create backup and migrate it to the kubernetes GitLab deployment
#### Create backup
1. Create backup:
gitlab-backup create
gitlab_settings_bkp_file="/var/opt/gitlab/backups/settings-dump-$(date +%Y%m%d).tgz" tar -czf $gitlab_settings_bkp_file /etc/gitlab
2. Save backup from `/var/opt/gitlab/backups/` (host path: `$GITLAB_HOME/data/backups/`) somewhere save and accessable
#### Deploy new cp-gitlab-db pod
1. Create a new folder for `gitlab db` on the system share or local folder on a mater node which hosts all deployment dirs, f.e:
mkdir /opt/gitlab-17-postgresql/data
```
deploy/contents/k8s/cp-gitlab-db/cp-gitlab-db-dpl.yaml
and deploy/contents/k8s/cp-gitlab-db/cp-gitlab-db-svc.yaml
create fresh gitlab-db
deployment
cp-gitlab-db
-> cp-gitlab-db-17
)volumes/gitlab-pgdata
from /opt/gitlab-postgresql/data
to /opt/gitlab-17-postgresql/data
kubectl apply -f <file>
Inside the gitlab-db container execute:
psql -U postgres -c "create user gitlab createdb;"
psql -U postgres -c "alter user gitlab with superuser;"
psql -U postgres -c "alter user gitlab with password 'gitlab';"
psql -U postgres -c "create database gitlabhq_production owner gitlab;"
gitlab
on the system share or local folder on a mater node which hosts all deployment dirs, f.e:
mkdir -p /opt/gitlab-17/logs
mkdir -p /opt/gitlab-17/config
mkdir -p /opt/gitlab-17/data
mkdir -p /opt/gitlab-17/pki
cp -r /opt/gitlab/pki/* /opt/gitlab-17/pki/
deploy/contents/k8s/cp-git/cp-git-dpl.yaml
create fresh gitlab
deployment
cp-git
-> cp-git-17
)volumes/git-logs
from /opt/gitlab/logsto
/opt/gitlab-17/logs. The same for volumes
volumes/git-config,
volumes/git-data`volumes/git-pki
containers/env
section:
env:
kubectl apply -f <file>
After appliyng:
Copy prepared backup
cp <backup location>/<backup>_gitlab_backup.tar /opt/gitlab/data/backups/
chmod a+r /opt/gitlab-17/data/backups/<backup>_gitlab_backup.tar
Restore backup
kubectl exec -it $GITLAB_POD gitlab-ctl stop puma
kubectl exec -it $GITLAB_POD gitlab-ctl stop sidekiq
kubectl exec -it $GITLAB_POD gitlab-rake gitlab:backup:restore BACKUP=$GITLAB_BACKUP
--- yes
--- no
Copy configuration files from settings-dump backup (except for gitlab.rb) "cp
Reconfigure gitlab
kubectl exec -it $GITLAB_POD gitlab-ctl reconfigure
Restart git pod and check status
kubectl exec -it $GITLAB_POD gitlab-rake gitlab:check SANITIZE=true
Background It would be great to have possibility to deploy CloudPipeline with fresh GitLab version (17.4.1 as of now)