Closed mihelm closed 6 months ago
I also encountered the same problem
cat /proc/version
Linux version 3.10.108 (root@build6) (gcc version 12.2.0 (GCC) ) #69057 SMP Sat S
ep 23 21:44:56 CST 2023
docker version
Client: Version: 20.10.23 API version: 1.41 Go version: go1.19.1 Git commit: 876964a Built: Fri Sep 8 08:19:48 2023 OS/Arch: linux/amd64 Context: default Experimental: true
Server: Engine: Version: 20.10.23 API version: 1.41 (minimum version 1.12) Go version: go1.19.1 Git commit: 5fcb1cf Built: Fri Sep 8 08:18:18 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.6.15 GitCommit: b23a389d8c181697302d163356e97dec04eb8d88 runc: Version: v1.1.4 GitCommit: 5af893d docker-init: Version: 0.19.0 GitCommit: ed96d00
I have same issue running 16.7.0. The logs for the gitlab service container:
gitlab_1 | rake aborted!
gitlab_1 | failed to get urandom
gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support.rb:26:in `require'
gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8/lib/active_support.rb:26:in `<top (required)>'
gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails.rb:7:in `require'
gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8/lib/rails.rb:7:in `<top (required)>'
gitlab_1 | /home/git/gitlab/config/application.rb:6:in `require'
gitlab_1 | /home/git/gitlab/config/application.rb:6:in `<top (required)>'
gitlab_1 | /home/git/gitlab/Rakefile:9:in `require'
gitlab_1 | /home/git/gitlab/Rakefile:9:in `<top (required)>'
gitlab_1 | /home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
gitlab_1 | /usr/local/bin/bundle:25:in `load'
gitlab_1 | /usr/local/bin/bundle:25:in `<main>'
I did some more digging and I'm fairly certain this is due to old linux kernels.
Since I'm running GitLab on a Synology, there's nothing I can to about running a rather prehistoric 3.10 kernel.
Short of moving GitLab to a different system I guess the only option would be to run it on Docker inside a VM on the same Synology. I don't much fancy the performance overhead of that approach. Anybody got practical experience with that?
Have the same issue with gitlab on Synology. It seems the only hope is when Synology update one day the gitlab package which is based on this repo / docker image.
@Algorab I was under the impression that Synology made clear they wouldn't be releasing any more GitLab updates. I had a quick look, but couldn't find any source on that, so I could be wrong. But even if they release another GitLab update, I'm pretty sure that's not the issue here. 16.7 requires Ruby 3.1. And Ruby 3.1 doesn't work with older Linux kernels, like the 3.10 Linux kernel found on any Synology that's at least 5 years old or so. Inexplicably, Synology make it a policy not to update the Linux kernel of their storage products, except for the occasional back-port of critical security updates. Unless I'm missing something, that means that if a Synology can't run GitLab 16.7 today, then it never will, except perhaps in a VM.
Just found myself in the same situation. I finally migrated my GitLab Setup via Synology Package to an Ansible & Docker-Compose Setup on my 916+ because this package blocked me to actually upgrade from DSM 6.2 to 7.x. I went the whole upgrade path 13.12.2 => 14.0.6 => 14.3.3 => 14.9.3 => 14.10.5 => 15.0.3 => 15.4.6 => 15.11.13 => 16.1.5 => 16.3.6 => 16.7.3 => just to see that my gitlab won't start after my last upgrade because of this kernal issue and that Synology might not solve this.
Still I ask myself, where this urandom method request was introduced. On the GitLab code or in the underlying OS Image. I really don't want to host my Gitlab on other hardware.
@CDann3r I think you should be able to just start it with 16.6.5 — 16.7 never comes up and never even gets started with any migrations. So your database should still be in the state it was in when 16.3.6 was running. But chances are you won't be able to upgrade GitLab, except for the remaining updates to 16.6 that will presumably be released over the next moth or so.
@mihelm That is what I did the last hour and thanks god it works. Thank you that you reported it. It helped me to find a quick solution. Not being able to upgrade is what worries me, that there will be one critical CVE that won't be patched on 16.6.x. Maybe it is worth opening an issue on GitLab directly to add compatibility or at least update the system requirement documentation. IMHO: The issue can't be solved in this repo.
I too have this problem running GitLab with docker-compose on a Synology RS1219+. Keeping Gitlab on 16.6.5 for now. But the admin page is getting angrier every time a new update gets released. But I fail to understand why #2886 is only a problem for Synology users (it seems). Wouldn't this affect all Docker users regardless of system? Or rather; all Omnibus users.
Wouldn't this affect all Docker users regardless of system? Or rather; all Omnibus users.
No, I have the same issue and Docker containers do need the kernel to run. The issue seems to be that it's the old kernels used in Synology systems.
Well, it's not so much a GitLab issue as a Ruby issue — GitLab 16.7 requires a newer version of Ruby, which introduced the problem. It's conceivable a newer version of Ruby could fix it. But I have no idea if GitLab 16.7 would work with a newer version of Ruby than the one specified. It might though — the release notes of 16.7 state:
Administrators installing from source must have Ruby 3.1 as a minimum version when upgrading to GitLab 16.7 or later...
I don't think it's a Ruby issue. It's a GitLab issue. Before 16.7.0 I'm guessing GitLab didn't use the Random.urandom function from Ruby. I tried running these Ruby versions in docker containers on my Synology: 3.0.6, 3.1.4, 3.2.0, 3.2.3 and 3.3.0. Every one of them threw an error when I tried to run Random.urandom(1) in irb. On my Windows machine this didn't happen. The documentation for Random.urandom also states that it's a platform feature. I made a small ruby script overriding urandom:
class Random
def self.urandom(size)
Random.bytes(size)
end
end
I mapped this script to the container and defined IRBRC to point to it. This way I could call Random.urandom on the Synology.
If we could somehow make sure that this override would be present in the environment GitLab is running in, this might solve it. I was thinking of making a ruby alias (alias ruby=ruby -r /path/to/filemapping) but where to put it? I want to try /etc/profile.d/ but I'm not sure GitLab runs in a way that reads the /etc/profile... any ideas?
I don't think this issue is solvable in any other way than with a newer Linux version... so a fix like this one is probably the only way. And then I don't have the first clue about the consequences of overriding a function that provides "cryptographically secure pseudo-random number in binary form" ¯\(ツ)/¯
@Peter-Oestergaard I don't think so. The error occurs while Ruby/Rails is being initialised — I think when active_support
checks its requirements:
https://github.com/rails/rails/blob/main/activesupport/lib/active_support.rb#L26
This isn't GitLab making a strange call. This is one of the gems
Ruby tries to load tripping up.
@mihelm You are of course entitled to your own opinion. I did however just succeed in upgrading from 16.6.5 to 16.7.3 to 16.7.4 to 16.8.1 by mapping a modified securerandom.rb file where i changed Random.urandom calls to Random.bytes calls.
I wouldn't exactly call this a bulletproof solution as I will now have to keep Ruby versions in mind whenever GitLab updates as I will then need to reflect any changes to securerandom.rb in my own version... you can't win them all.
Could you please elaborate in detail, how your solution can be implemented? That would be really great - I have running GitLab in Docker on my Synology and have the same problem...
@Peter-Oestergaard Wouldn't you say that by injecting a patch for ruby/lib/securerandom.rb
as a workaround, you have just proven that this is in fact a Ruby issue?
Incidentally, securerandom
is the very dependency being checked on the line of code I indicated in my last.
Your workaround, together with what was posted in #2886, makes me think that building 16.7 with Ruby 3.2 would be worth a try.
@mihelm There are multiple reasons why I still think this is a change in GitLab (is this part of GitLab open source? If it is it should be trivial to confirm):
@mbuchber Sure thing. It really doesn't take much:
/opt/gitlab/embedded/lib/ruby/3.1.0/securerandom.rb
I did encounter an error about securerandom not being found. The error was not "file not found", but something like unable to load. This was because the file inside the container had the wrong settings. I did a chown root:root
and a chmod 644
on it on the docker host to solve it.
@mihelm Ok. I am ready to concede. I took a look at the GitLab source. Nowhere are calls made to urandom directly. I found a lot of calls to the other methods in the SecureRandom module. But nothing that could back my guess that GitLab made any changes. So I went back in to the different Ruby containers on the Synology. And I see what you mean with regards to the dependency check as the error is reported in 3.1.4 when calling require 'securerandom'
. What is of some interest is that doing the same on 3.2.0 and 3.2.3 is not failing, but a subsequent call to SecureRandom.hex(1) is. But on 3.3.0 it's all working again.
TLDR: Random.urandom will always fail on the Synology, but that function has never been used. Functions in SecureRandom will fail on versions 3.1.4, 3.2.0 and 3.2.3 and that is a problem since those are functions that GitLab uses.
But maybe when GitLab bumps Ruby to version 3.3.0 I can disable my fix again...
@Peter-Oestergaard Unfortunately, GitLab still crashes with Ruby 3.2 on Linux kernel 3.10, just much later during startup. @kkimurak created PR #2893 to upgrade Ruby. I built a 16.7.4 image from his 16.7.x branch and tried it on my Synology. The container no longer crashed right at the start — but it did crash, just later while trying to initialise invisible_captcha.rb
.
It was worth a try, alas it didn't work.
I doubt adapting current versions of Ruby (and its libraries) to work with such ancient kernels is high up the list of priorities for the Ruby project.
Consequently, that leaves your workaround as the only option for getting GitLab to work on these old kernels.
There is one thing that everybody using @Peter-Oestergaard 's instructions should keep in mind: Since the path on which the modified securerandom.rb
is mounted contains the version number of Ruby, this will need to be tweaked every time the Ruby version included in the container is updated.
That's not a problem per se, just something to remember on future updates.
According to the backtrace reported by @mihelm (https://github.com/sameersbn/docker-gitlab/pull/2893#issuecomment-1912663336), the strange thing is gen_random_openssl
uses urandom
even it possibly not supported.
gitlab | /usr/local/lib/ruby/3.2.0/securerandom.rb:55:in `urandom' gitlab | /usr/local/lib/ruby/3.2.0/securerandom.rb:55:in `gen_random_openssl' gitlab | /usr/local/lib/ruby/3.2.0/random/formatter.rb:74:in `random_bytes' gitlab | /usr/local/lib/ruby/3.2.0/random/formatter.rb:94:in `hex' gitlab | /home/git/gitlab/vendor/bundle/ruby/3.2.0/gems/invisible_captcha-2.1.0/lib/invisible_captcha.rb:46:in `init!'
This behavior have been fixed on upstream (https://github.com/ruby/ruby/commit/64e503eb62aff0952b655e9a86217e355f786146) but applied only for 3.3.x release.
One of the choise is to apply this patch on building Ruby during image build.
@kkimurak I tried building the image with Ruby 3.3.0, but the build failed — the API has drifted too far, and at least one needed command wasn't available anymore.
I'm wondering if patching bugs in Ruby during build isn't starting to stray beyond the scope of this project... Are there are instances where this sort of thing is done?
@mihelm
I'm wondering if patching bugs in Ruby during build isn't starting to stray beyond the scope of this project...
If you think so I will just stop working on this issue. I'm ok with it because I don't use Synology.
However, Synology users here (including you) are not just loudly demanding support, they are investigating what the underlying problem is. I would like to help them as far as I can.
Are there are instances where this sort of thing is done?
This project applies some patches to gitlab on build. GitLab itself applies some patches for dependencies like rails/ujs, vue-jest and so on.
I'm just a contributor so I just write a patch. If the maintainer think that such a patch is not needed for this project, it will be rejected. That's all.
I tried building the image with Ruby 3.3.0, but the build failed
Maybe I got same result - failed on installing rails 7.0.8 (bundle install
for gitlab - the log shows that gitaly have been built successfully). It seems that rails version unmet with Ruby 3.3.x.
Here is a build log. I executed { time make release 2>&1; } | ts -s | tee build-$(cat VERSION).log
so elapsed time from build starts in %H:%M:%S format is added to the beginning of each line by ts -s
command.
Edit 2024-02-13 : as 16.8.2 and 16.7.5 released, I have removed these branches. If you really need to restore them, refer kkimurak@0398b16177d9efe266966e42ae9c91e25e13fe3f (based on 16.8.1) and kkimurak@025fff49aa21a880e59b5eb98b3770b5e4ac3df4 (based on 16.7.4).
By the way, I have pushed some branches that update ruby to 3.2.3 and backport https://github.com/ruby/ruby/commit/64e503eb62aff0952b655e9a86217e355f786146 . I have confirmed that it built successfully and at least it launches, sends notification mail (like admin password reset) and new blank project can be created from Web UI. Anyone want to try to run it on Synology ?
@kkimurak I'm happy to test that. Thanks for the work!
@kkimurak It worked! Took me a while to wangle the image over to the Synology, but once it was there and properly tagged, it came up without the slightest issue!
I tested the 16.7 branch:
I was worried the bug you fixed with the backport might just be the first to pop up. I'm glad I was wrong on that!
Awesome! unfortunately I have an issue with my image building process on my new hardware. When I tried to build the images then I couldn't even just start bash even though the base image was fine. So I wasn't able to test your suggestions at all. Still not sure where the issue is on my windows system but will try with the latest changes. If not I will come back and beg someone to upload their build on docker hub until this PR gets approved
Happy to see such a good progress. Thank you!
@kkimurak It worked! Took me a while to wangle the image over to the Synology, but once it was there and properly tagged, it came up without the slightest issue!
I tested the 16.7 branch:
![]()
I was worried the bug you fixed with the backport might just be the first to pop up. I'm glad I was wrong on that!
how did you do the upgrade using this branch please? Am on Synology also and gitlab 16.6.6 at the moment
@drguiom I built the image on a different system, then transferred it to the Synology, tagged it and edited my docker-compose.yml
to use it. I did this purely for testing purposes, and it took over an hour all told — and that was with a branch ready to go. I would not advise anyone to use this workflow for a GitLab server they intend to use. Not least because they'd probably have to assemble their own branch for future updates.
kkimurak has submitted PRs to merge the fix into this repository. I think your best hope is that the maintainers approve those PRs. Short of that, you could follow the instructions Peter-Oestergaard posted earlier. That workaround achieves the same result, but needs to be tweaked every time Ruby is updated in the image.
@drguiom I built the image on a different system, then transferred it to the Synology, tagged it and edited my
docker-compose.yml
to use it. I did this purely for testing purposes, and it took over an hour all told — and that was with a branch ready to go. I would not advise anyone to use this workflow for a GitLab server they intend to use. Not least because they'd probably have to assemble their own branch for future updates.kkimurak has submitted PRs to merge the fix into this repository. I think your best hope is that the maintainers approve those PRs. Short of that, you could follow the instructions Peter-Oestergaard posted earlier. That workaround achieves the same result, but needs to be tweaked every time Ruby is updated in the image.
ok many thanks for the prompt response, am upgrading the NAS soon so hopefully the kernel will have been updated. am fine on 16.6.6 and was upgrading "because I can" as opposed to needing to. Its internal use only anyway ;-)
I can confirm that the newly built image for 16.7.5 came up without any problems on my Synology. It took 5-6 minutes, which is a bit longer than it took for 16.6, but I guess that's just the CPU showing its age — GitLab launch times have been trending up on my Synology for a while now.
@kkimurak Thank you very much for figuring out the backport and for submitting those PRs! I'm sure that's helped a lot of people!
@mihelm Glad it works.
I would not have attempted to check the details if you had not tried to work out the details. I also didn't have the environment to look into it, so I am very grateful to you for checking the operation and reporting back to me on the actual device.
Thank you for the contribution!
I can confirm that the newly built image for 16.7.5 came up without any problems on my Synology. It took 5-6 minutes, which is a bit longer than it took for 16.6, but I guess that's just the CPU showing its age — GitLab launch times have been trending up on my Synology for a while now.
@kkimurak Thank you very much for figuring out the backport and for submitting those PRs! I'm sure that's helped a lot of people!
same here, great work!
now up to 16.8.2 on Synology NAS 415+ and running well
Same for me on a Synology 916+
I observed 16.7.5 for a day and everything works fine. Later on I will go to 16.8.
Hi
16.7.0 refuses to start for me. Rake aborts with "failed to get urandom".
Went back to 16.6.2, which came up without a hitch.
Any ideas what might be causing that?
I use docker-compose. Docker isn't a terribly recent version, and unfortunately I have no control over which version I can run.
Cheers, michael.