Database update/migration failed from v12.9.2-0055 to v13.0.3-0055 #44

Closed helmut-steiner closed 3 years ago

Hey, I ran the update from GitLab v12.9.2-0055 to v13.0.3-0055 early this morning but it seems like my database was not correctly migrated to the new postgresql version. After the update I was presented with a password reset screen not with my usual login screen and I even after that I couldn't login anymore with my username and password. I had to uninstall the new version, clear the files, reinstall the old version and reload my backups. It looked like it didn't port any of my settings over but the upgrade logs didn't show any abnormalities. Kind regards, Helmut

Hi Helmut,

thanks for your backup information. I've spent the whole afternoon to get the GitLab 13 working. But there is something wrong with the docker image. Although the migration seems to work sometimes from 12.9.2 to 13.0.3, the clean installation and migration to higher versions fails. I am not sure how to solve this without change everything and keep the compatibility.

The logical step would be a change to the official giltab-ce and postgres image. This could be curse or blessing.

I will keep you up to date.

I am going for a new package based on the official gitlab-ce postgres and redis image. This is a lot of work and i can't promise to get this done. I need to learn setup the gitlab-ce properly and combine this with the other images. I also aim to get this done with the official Synology toolkit to get this package signed. If this works out i will be able (hopefully) to get this package published in a repo so you can get auto updates.

wish me no luck, but patience and time :-)

This is good news. I thought about switching to the original packet myself a couple of times already.

Do you need help?

I also think this will be the best solution. The update never worked on our two year old NAS. So it should be a fast enough system.

I can also try to help if needed

Hello guys, this took me half a day to figure it out but now i have at least a diry "workaround".

There were two issues, one were the missing "btree_gist" and the second is the missing "ghost" column in the users table.

Today i solved the issue with the btree_gist together with @helmut-steiner.

But the missing column is still an issue. The Column has been removed with GitLab 13.4.0. I traced the missing column down to 13.0.6 so with the newer 13.2.6 the column is missing but the GitLab Codes seems still to use it. There are same issues out there.

As a Workaround we need to add the column.

Auto ADD: sudo /usr/local/bin/docker exec --user=postgres -it synology_gitlab_postgresql bash -c "psql -d gitlab -c 'ALTER TABLE users ADD COLUMN IF NOT EXISTS ghost BOOLEAN'"

Auto REMOVE (whene not needed anymore): sudo /usr/local/bin/docker exec --user=postgres -it synology_gitlab_postgresql bash -c "psql -d gitlab -c 'ALTER TABLE users DROP COLUMN ghost'"

Manual:

Docker App => Terminal => Create => bash
sudo -u postgres psql -d gitlab
ALTER TABLE users ADD COLUMN IF NOT EXISTS ghost BOOLEAN;
\q

@helmut-steiner could you give it a try. Just pull the current build script and add the column when you get the 500 on you login screen.

Unfortunately this will solve this specific issue, but there could be more of them because i could not find out why the column is missing but is still in use by the code. There seems to be a mismatch in code an schema version and there could be more columns missing or may have the wrong data type.

Hey Juri, If you are a signed in user the issue doesn't occur. That's why I didn't see it during my first tests as I still had my session cookie. I tried it on my test system and the workaround (auto add column) did the trick. The login screen works as expected.

Hi Helmut,

could you please do some more testing with your data set and see if you find something broken. If you do not find further issues i am thinking about to make a "experimental" release so everybody who feels save with backup and restore can give it a try.

Kind Regards

Yeah, I will play around a bit tomorrow and let you know. :)

Okay, I ran further tests:

Login works
Checkout/pull/push works
Admin area works as far as I tested it
Resetting runners in the DB works (if you forgot your secrets or didn't back them up...)
Creating/closing/modifiying/assigning issues works
Basic project/repository sites are accessible (graph, commits, wiki, analytics, settings, members...)

So far things look good. :)

Good work guys. I will help to test if we have an experimental release

Version 13.4.2 has been released by sameersbn. @jboxberger Have you tried if the issues still persist with that version? I can't build it with the script.

Hi @helmut-steiner, i could build it without any issues. What was the error? I've tested the 13.4.3 and the problem still persist. But i published a experimental release.

https://github.com/jboxberger/synology-gitlab/releases

I've also discovered a non breaking error during the update from 12.9.2 to 13.4.3 but i could no see any effects. Same problem here a missing column... :-1:

2020-10-09 13:01:37.004 UTC [143] ERROR:  column "default" of relation "services" does not exist at character 200

2020-10-09 13:01:37.004 UTC [143] STATEMENT:  WITH created_records AS (

      INSERT INTO services (project_id, "active","properties","type","template","push_events","issues_events","merge_requests_events","tag_push_events","note_events","category","default","wiki_page_events","pipeline_events","confidential_issues_events","commit_events","job_events","confidential_note_events", created_at, updated_at)

      SELECT "projects"."id", true,'{}','PrometheusService',false,true,true,true,true,true,'monitoring',false,true,true,true,true,true,true, TIMEZONE('UTC', NOW()) as created_at, TIMEZONE('UTC', NOW()) as updated_at FROM "projects" LEFT JOIN services ON services.project_id = projects.id AND services.project_id BETWEEN 1 AND 1

                        AND services.type = 'PrometheusService' INNER JOIN cluster_groups ON cluster_groups.group_id = projects.namespace_id INNER JOIN clusters_applications_prometheus ON clusters_applications_prometheus.cluster_id = cluster_groups.cluster_id

                          AND clusters_applications_prometheus.status IN (3, 5) WHERE (projects.id BETWEEN 1 AND 1 AND services.id IS NULL)

      RETURNING *

    )

    SELECT COUNT(*) as number_of_created_records

    FROM created_records

That's the thing: I don't even get an error. It just doesn't do all the things it is supposed to. It doesn't download the images and just builds the final file which is only 560kb big.

============================================================================================================
 Please provide the full qualified docker container name from dockerhub.com or version number
============================================================================================================
GitLab [default sameersbn/gitlab:13.4.3]: 13.4.3
Postgres [default sameersbn/postgresql:11-20200524]:
Redis [default redis:5.0.9]:
export image sameersbn/gitlab:13.4.3
  100 %               240 B / 203 B = 1.182
export image sameersbn/postgresql:11-20200524
  100 %               240 B / 203 B = 1.182
export image redis:5.0.9
  100 %               240 B / 203 B = 1.182
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  540k  100  540k    0     0   861k      0 --:--:-- --:--:-- --:--:--  859k

Btw: Are we using postgresql 11 or 12 now?!

The 13.4.3 Version doesn't work for me. After installation, I get a 502 error. And if I add the ghost column, it works, but my credentials don't. Also, my custom login screen is lost.

@iConMayrhofer Sounds like the problem I had with my old NAS where the CPU was too slow and the DB migration got into a timeout. Which NAS are you using?

DS1517+

Hello @iConMayrhofer,

the advice of Helmut might be correct. As far as i understood the Gitlab DB Upgrade procedure happens incremental. So by restarting this will continue where it stopped. But not if there were a other error in between. You need to check the log files and search for errors. Please post them here, maybe we can help.

grafik

Kind Regards

I've got good news. I got the Synology toolkit running and i got the gitlab-ce running, i like this solution way more than the synology-gitlab approach. I will aim for no need for spk, but will offer a very thin installer for beginners.

@helmut-steiner : do you want to give it a try? This can run absolutely parallel to the current installation. You just need to create the folder where the Gitlab files should be located.

Please rename the gitlab-ce-13.4.3-ce.0.json.txt => gitlab-ce-13.4.3-ce.0.json gitlab-ce-13.4.3-ce.0.json.txt

create: /docker/gitlab-ce/data /docker/gitlab-ce/logs /docker/gitlab-ce/config

grafik

Pull docker image: grafik

Import json in docker: grafik

grafik

start container: grafik

After the container booted up (first time takes a while, ready when about 2GB RAM usage) you can access it via http://your-host-or-ip:30080

The great about this solution, only one container, exposed config and logfiles (be careful with the permissions).

Could you please test if you can migrate your data into gitlab-ce by simply export it from your current installation and import them into the gitlab-ce?

Kind Reagrds

I've got good news. I got the Synology toolkit running and i got the gitlab-ce running, i like this solution way more than the synology-gitlab approach. I will aim for no need for spk, but will offer a very thin installer for beginners.

I think this is the same approach as documented by Matthias Lohr on his website. I switched myself to that approach due to the issue here. As far as I can see it works fine, how ever the CPU load is higher than with your spk-approach, see https://mlohr.com/gitlab-on-a-diskstation/#comment-33301

@jboxberger and @helmut-steiner can you confirm this?

@jboxberger I will give it a try!

@jboxberger how to import into gitlab-ce? Thanks

@fsutter

1) Copy your backup file to docker\gitlab-ce\data\backups. 2) From the official Gitlab documentation:

# Stop the processes that are connected to the database
sudo docker exec -it <name of container> gitlab-ctl stop unicorn
sudo docker exec -it <name of container> gitlab-ctl stop puma
sudo docker exec -it <name of container> gitlab-ctl stop sidekiq

# Verify that the processes are all down before continuing
sudo docker exec -it <name of container> gitlab-ctl status

# Run the restore
sudo docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce

# Restart the GitLab container
sudo docker restart <name of container>

@helmut-steiner I've also found these instructions but I got this error message: No backups found in /var/opt/gitlab/backups Please make sure that file name ends with _gitlab_backup.tar

How do I change the default backup directory? I haven't the /var/opt/gitlab directory on my synology

Thanks

@fsutter Copy the backup file in the data\backups folder in your gitlab-ce folder.

@jboxberger I installed the gitlab-ce 13.0.3 version and restored my production backup there. It works fine. Now I am looking into how to get HTTPS going as letsencrypt doesn't have access to the special ports. I am thinking of copying the letsencrypt certs from the NAS to the gitlab-ce installation. Thoughts on that for a better way as this has to be done manually after each renewal?

Hi, it worked also for me. Now I have to figure out how to "configure" it. I'm using an external reverse proxy and with the synology package I used GITLAB_HTTPS, GITLAB_HOST and GITLAB_PORT environment variables to specify how to access from external. I've also used GITLAB_BACKUP_EXPIRY to rotate backups.

Frédéric

Hey there, I wrote a script to copy the certificate and key files into the ssl folder under config: 1) Create a "ssl" folder in gitlab-ce/config for user root:root with access rights 755 2) Create a securemygitlab.sh file with following content and replace the gitlab.yourdomain.com with your gitlab domain:

#!/bin/bash
echo Copying TLS certificates to gitlab...
cd /usr/syno/etc/certificate/system/default
echo Copy new gitlab.crt
rm /volume1/docker/gitlab-ce/config/ssl/gitlab.yourdomain.com.crt
cat fullchain.pem cert.pem > /volume1/docker/gitlab-ce/config/ssl/gitlab.yourdomain.com.crt
echo Copy new gitlab.key
rm /volume1/docker/gitlab-ce/config/ssl/gitlab.yourdomain.com.key
cp privkey.pem /volume1/docker/gitlab-ce/config/ssl/gitlab.yourdomain.com.key
echo Done.

3) sudo ./securemygitlab.sh

All set for https! :D

Be sure to set following variables in gitlab.rb external_url 'https://gitlab.yourdomain.com'

Also set letsencrypt to false: letsencrypt['enable'] = false

You can also use the script in the task scheduler of the NAS to execute every couple of days so your gitlab instance will always have an up-to-date certificate. :)

Aaaand finally: Update from gitlab-ce 13.0.3 to 13.4.4 worked. :) I will play around a bit but except the higher CPU usage so far nothing speaks against production usage.

@helmut-steiner I tried to change external_url but after gitlab is no more responding :( I tries either by passing GITLAB_OMNIBUS_CONFIG in the .json file or by modifying gitlab.rb, same problem.

Thanks

@fsutter Did you restart the docker container after changing the URL? Is your Gitlab URL different than your NAS URL? If so the above script doesn't work.

@helmut-steiner Yes, it's precisly when restarting that gilab never respond even after 20 minutes. If I put the line in comment it works again. I haven't tried the script yet.

ragards,

@fsutter If your URL contains https then you have to add the certificate and key file before starting up the container. Otherwise it won't work.

@helmut-steiner I'm using a resverse proxy that makes the http->https "translation". How do I deal with that?

@fsutter Sorry I haven't looked into that yet. I might have time on the weekend to experiment with setting up the reverse proxy. There is some information on this blog posts and in the comments on how to get reverse proxy working: https://mlohr.com/gitlab-on-a-diskstation/

@fsutter I am not into reverse proxies, but could you please have a look here. Maybe this helps. I am also very interested in the solution, if you get this running please post how, i will add it later for a wiki.

https://www.itsfullofstars.de/2019/06/gitlab-behind-a-reverse-proxy/

external_url 'https://gitlab.itsfullofstars.de'

gitlab-ctl reconfigure

@caco3: Thank you for your link. I haven't knew that it's that easy since the sameersbn image is really complicated i could not really get into it. I started the project around Gitlab 8.x.x, since then some things changed an i grew with the project. I started as an absolute novice in Gitlab and Docker so i took something working and improved it, but now i am a bit further.

Once everything is up and running a will make a thin installer for beginners. This thin installer aims for people who are new to Synology, Docker, Gitlab and just want to get it running. I want it to be fully compatible to the Manual configuration so you can later simply switch.

Kind Regars

Okay. I tried it and the reverse proxy works flawlessly with my setup. Here are my settings: Screenshot 2020-10-16 142307 Screenshot 2020-10-16 142414

Hi,

it works by adding this line to the .json: { "key" : "GITLAB_OMNIBUS_CONFIG", "value" : "external_url 'https://my.url.com'; nginx['listen_port'] = 80; nginx['listen_https'] = false" }

Thanks for the support and kind regards,

Frédéric

Hi,

it works by adding this line to the .json: { "key" : "GITLAB_OMNIBUS_CONFIG", "value" : "external_url 'https://my.url.com'; nginx['listen_port'] = 80; nginx['listen_https'] = false" }

Thanks for the support and kind regards,

Frédéric

Ah, so you are proxying from https to an internal http server. Yeah. I put the requests right through - that's why I need the certificates also inside the container.

Yes, I'm using HAProxy on pfsense.

Other problem: I did a backup from 12.9.2 gitlab-synology, installed gitlab-ce, restored and everything worked fine (except Web IDE but that's another story). Then I deleted the container without removing /docker/gitlab-ce share and tried gitlab-ce-latest but it doesn't work, container will not start :(

I got one more question as well: How do I set the SSH port to show the outside port 30022 or the formerly used 30001 inside gitlab instead of 22 which is used internally?

If I set gitlab_rails['gitlab_shell_ssh_port'] = 3022 it also uses the 30022 port internally. Well maybe I just change the environment variables... Thoughts on that?

I got one more question as well: How do I set the SSH port to show the outside port 30022 or the formerly used 30001 inside gitlab instead of 22 which is used internally?

If I set gitlab_rails['gitlab_shell_ssh_port'] = 3022 it also uses the 30022 port internally. Well maybe I just change the environment variables... Thoughts on that?

Never mind. setting the gitlab_rails['gitlab_shell_ssh_port'] = 3022 works just fine. There was something wrong with my port forwarding. 🤦‍♂️

Yes, I'm using HAProxy on pfsense.

Other problem: I did a backup from 12.9.2 gitlab-synology, installed gitlab-ce, restored and everything worked fine (except Web IDE but that's another story). Then I deleted the container without removing /docker/gitlab-ce share and tried gitlab-ce-latest but it doesn't work, container will not start :(

@fsutter Do you have any error log or error message?

@helmut-steiner no sorry, due to my Web IDE issues, I changed strategy: I made an export of all my projects, installed gitlab-ce-latest and re imported all projects into it. Now all works fine.

regards,

Frédéric

Upgrade to the gitlab-ce:13.4.4-ce.0 image worked. 👍

I found out that in version 13.4.4 three of my repos were empty. I could fix this by restoring to version 13.0.3 and then manually doing the hashed storage miration: sudo docker exec -it gitlab-ce gitlab-rake gitlab:storage:migrate_to_hashed

After that I could update to 13.4.4 without any problems. I think the problem is somewhere in a version higher than 13.3, because until then I could update without problems.

@jboxberger Thanks for all the time you put into this. Will you provide a new/fixed package in the near future?

It looks like there's new stock synology package with version 13.6.2. https://www.synology.com/en-global/dsm/packages/Docker-GitLab

I don't think that they will be updating it regularly, so I would be very grateful if this can be updated to recent version.

@crux153

Hi thanks for your hint. I played a bit around and it look good. Although i need to test the migration path from the previous versions. If you have the ability to test without risking your production data, i can provide you a test version.

Kind Regards

Hello my friends, thank you for your patience... i've did a whole rewind of the project and tested a dozen of versions (not joking > 20h in total) to get a stable upgrade path. Please give me a few more days to cleanup and upload the packages.

@crux153: since you're on the experimental package, you need specialized version to get a clean migration. I will test the migration path for you so you have as less risk as possible. Please do Backups and check if you can restore them anywhere in a test environment just to prevent bad surprises. Your upgrade path is the risky one but i will help as good as i can.

Upgrade for 13.0.3-0055: 13.0.3-0055 => 13.0.3-0058 => 13.6.2-0068 Upgrade for 13.4.3-0055: 13.4.3-0055 => 13.4.3-0058 => 13.6.2-0068

@caco3: yes they will the problem were the migration of PSQL in the sameersbn gitlab image, it looks like it is solved by now so i can go on regular.

Preview:

jboxberger / synology-gitlab

Database update/migration failed from v12.9.2-0055 to v13.0.3-0055 #44