Closed toupeira closed 2 years ago
kwilczynski
commented
2 years ago Hi @toupeira,
Sorry for the issues! Let me check what is going on. I'd hope that Scaleway (one of the mirrors, and this particular one) wouldn't cause an issue. Bear with me.
Krzysztof
kwilczynski
commented
2 years ago Hi @toupeira,
I had a look, and it seems to be fine. I see an increase in traffic, most likely due to update of dependencies in GitLab, however, I sadly don't have a lot of visibility into potential issues as there is next-to-nothing in terms of logging available on Scaleway (or I can't find it).
Are you having this problem continuously?
Krzysztof
toupeira
commented
2 years ago @kwilczynski thanks for checking! I do get the problem continuously, but I tried it on a different machine now and there it works, because it's connecting to a different IP:
$ wget https://ruby-magic.s3.nl-ams.scw.cloud/file-5.39.tar.gz
--2022-01-12 13:54:28-- https://ruby-magic.s3.nl-ams.scw.cloud/file-5.39.tar.gz
Resolving ruby-magic.s3.nl-ams.scw.cloud (ruby-magic.s3.nl-ams.scw.cloud)... 163.172.208.8, 2001:bc8:1401::8
Connecting to ruby-magic.s3.nl-ams.scw.cloud (ruby-magic.s3.nl-ams.scw.cloud)|163.172.208.8|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 954266 (932K) [application/x-gzip]
Saving to: ‘file-5.39.tar.gz’
file-5.39.tar.gz 100%[==============================================================================>] 931.90K --.-KB/s in 0.05s
2022-01-12 13:54:28 (17.8 MB/s) - ‘file-5.39.tar.gz’ saved [954266/954266]On the machine where it's failing it's resolving the hostname to 195.186.208.193, so I guess that mirror is having problems.
kwilczynski
commented
2 years ago Hi @toupeira,
Thank you for looking into this some more! I appreciate that.
I did a quick DNS resolution test and if Scaleway, indeed, did some DNS cur-over/change, then it seems that a lot of places already expired their cache as per:
I also seem to resolve it to the 163.172.208.8 address:
The other IP address seems to be in an odd (as I would imagine being under Scaleway, rather) network as per:
inetnum: 195.186.208.0 - 195.186.211.255 netname: BLUEWININFRA-NET descr: Swisscom (Schweiz) AG country: CH admin-c: BCR1-RIPE tech-c: BCR1-RIPE status: ASSIGNED PA remarks: INFRA-AW remarks: In case of hack attacks, spam, scans etc. please remarks: send abuse notifications to: abuse@bluewin.ch remarks: E-Mails to the persons below will be IGNORED! mnt-by: BLUEWINNET-MNT created: 2017-12-12T10:10:01Z last-modified: 2017-12-12T10:10:01Z source: RIPE # Filtered
And it does send garbage back:
I wonder if there is some stale DNS cache somewhere where your machine is located. What do you think?
Krzysztof
toupeira
commented
2 years ago Aahhh.... Swisscom is my ISP, so now I tried loading http://195.186.208.193/ directly (without HTTPS) and see this:
I keep forgetting that they have this blocker since I rarely run into it :sweat_smile: I reported this as a false positive now and hope it will be resolved soon, and if not I'll just switch to a sane DNS server :wink:
Thanks for your help, I'll close this issue now since it's clearly a problem on my end.
kwilczynski
commented
2 years ago Hi @toupeira,
Ah nice find! Admittedly, I haven't tried HTTP while doing some cursory look, sorry about that 😅
I am glad everything worked out in the end! Let me know if you need anything else. I hope you are going to enjoy new GitLab release.
Krzysztof
toupeira
commented
2 years ago No worries, thanks again! :grinning:
xperseguers
commented
2 years ago Hi there, I have the exact same problem trying to upgrade my GitLab instance.
I could successfully manually install ruby-magic (compiling it with --use-system-library) locally but each time I run
sudo -u git -H bundle installit fails with:
Using rqrcode 0.7.0
Using rqrcode-rails3 0.1.7
Using ruby-fogbugz 0.2.1
Fetching ruby-magic 0.5.3
Installing ruby-magic 0.5.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/ruby-magic-0.5.3/ext/magic
/usr/local/bin/ruby -I /usr/local/lib/ruby/site_ruby/2.7.0 -r ./siteconf20220127-15305-1n9mi8b.rb extconf.rb
Building ruby-magic using packaged libraries.
Static linking is enabled.
Cross build is disabled.
Using mini_portile version 2.6.1
---------- IMPORTANT NOTICE ----------
Building Ruby Magic with a packaged version of libmagic-5.39.
Configuration options: --disable-silent-rules --disable-dependency-tracking --enable-fsect-man5 --disable-shared --enable-static CFLAGS\=-fPIC
2 retrie(s) left for file-5.39.tar.gz
1 retrie(s) left for file-5.39.tar.gz
0 retrie(s) left for file-5.39.tar.gz
SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/local/bin/$(RUBY_BASE_NAME)
--help
--clean
--enable-system-libraries
--disable-system-libraries
--use-system-libraries
--enable-static
--disable-static
--enable-cross-build
--disable-cross-build
/usr/local/lib/ruby/2.7.0/digest.rb:50:in `initialize': No such file or directory @ rb_sysopen - /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/ruby-magic-0.5.3/ports/archives/file-5.39.tar.gz (Errno::ENOENT)
from /usr/local/lib/ruby/2.7.0/digest.rb:50:in `open'
from /usr/local/lib/ruby/2.7.0/digest.rb:50:in `file'
from /usr/local/lib/ruby/2.7.0/digest.rb:35:in `file'
from /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:320:in `verify_file'
from /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:76:in `block in download'
from /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:74:in `each'
from /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:74:in `download'
from /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/mini_portile2-2.6.1/lib/mini_portile2/mini_portile.rb:175:in `cook'
from extconf.rb:163:in `block in process_recipe'
from extconf.rb:76:in `tap'
from extconf.rb:76:in `process_recipe'
from extconf.rb:287:in `<main>'
extconf failed, exit code 1
Gem files will remain installed in /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/ruby-magic-0.5.3 for inspection.
Results logged to /home/git/gitlab/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/ruby-magic-0.5.3/gem_make.out
An error occurred while installing ruby-magic (0.5.3), and Bundler cannot continue.
Make sure that `gem install ruby-magic -v '0.5.3' --source 'https://rubygems.org/'` succeeds before bundling.
In Gemfile:
ruby-magicI tried to manually download file-5.39.tar.gz and place it at the location it is searched for but it looks like everything is cleared again prior to trying to install and I'm a bit stuck.
I changed my DNS resolver from 9.9.9.9 to 1.1.1.1, but cannot figure out either what's the URL it's trying to fetch file-5.39.tar.gz as I don't see anything in the logs apart from "trying to fetch" but without further debugging info.
That server is behind a Swisscom connection as well, but since the DNS resolver is not the one from Swisscom, it shouldn't be a problem.
Anyway, some help on how to proceed would be great.
toupeira
commented
2 years ago @xperseguers I see your SSL error is different:
SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)Mine was caused by my ISP responding with HTTP to a HTTPS request:
SSL_connect returned=1 errno=0 state=error: (null)Maybe one of the Scaleway mirrors does indeed have an expired certificate? /cc @kwilczynski
You could also check what the hostname ruby-magic.s3.nl-ams.scw.cloud resolves to, it should be an IP with a WHOIS record from Scaleway (163.172.208.8 in my case).
xperseguers
commented
2 years ago Indeed:
$ dig ruby-magic.s3.nl-ams.scw.cloud
; <<>> DiG 9.11.5-P4-5.1+deb10u6-Debian <<>> ruby-magic.s3.nl-ams.scw.cloud
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6461
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;ruby-magic.s3.nl-ams.scw.cloud. IN A
;; ANSWER SECTION:
ruby-magic.s3.nl-ams.scw.cloud. 1800 IN CNAME s3.nl-ams.scw.cloud.
s3.nl-ams.scw.cloud. 86400 IN A 163.172.208.8
;; Query time: 84 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Jan 27 16:12:42 CET 2022
;; MSG SIZE rcvd: 89@xperseguers ah interesting, you get the same IP but I don't get a certificate error here (both with curl and Firefox).
Maybe there's an issue with the root certificates in your system? Looks like you're on Debian too, check if your ca-certificates package is up to date. You can also see more details about the certificate error with openssl s_client -connect ruby-magic.s3.nl-ams.scw.cloud:443.
xperseguers
commented
2 years ago @toupeira looks OK and my system is kept up-to-date, but could it be related to that I still somehow have that damn old Let's Encrypt root certificate?
$ openssl s_client -connect ruby-magic.s3.nl-ams.scw.cloud:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = s3.nl-ams.scw.cloud
verify return:1
---
Certificate chain
0 s:CN = s3.nl-ams.scw.cloud
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFgDCCBGigAwIBAgISA3tr7ySiOnuZUgs1tf0SCZ8FMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTExMjkxMzUyMTZaFw0yMjAyMjcxMzUyMTVaMB4xHDAaBgNVBAMT
E3MzLm5sLWFtcy5zY3cuY2xvdWQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDcbMOnEP3xbGXswj2LacogulbyPDR6dfKOQXSQONUPfst9i1YkzYIaFNw9
XpXbUP52lUFprN2ypEXMFaJRW2+7915cJV/QYdwdrVWltyBi2LCwDh3BvM7U0wxM
Hh39lQTtOZTlHORMKOAx4P73FMa+ManMYeRVG4F+bqstHTMGzQlrPZHtfv8pGIrf
XEWxLrE5//xKSdVHFegG5brO/UEzMtw4tH0gWPcLBD8oaDpEnoMc4rhabsqDmU1Z
erAs/7UKK8bXsgyK2Az+OcDHVmDKk4yu5XHFxcDamGLG3RTuBTiXqntJC8YITsYv
Yd4vIeV+YwsQ96AGWlBgJV5S5EgnAgMBAAGjggKiMIICnjAOBgNVHQ8BAf8EBAMC
BaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAw
HQYDVR0OBBYEFP/9sZ4C0Wn/JrnUAHFDpIHayrosMB8GA1UdIwQYMBaAFBQusxe3
WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0
cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5j
ci5vcmcvMHEGA1UdEQRqMGiCHSouczMtd2Vic2l0ZS5ubC1hbXMuc2N3LmNsb3Vk
ghUqLnMzLm5sLWFtcy5zY3cuY2xvdWSCG3MzLXdlYnNpdGUubmwtYW1zLnNjdy5j
bG91ZIITczMubmwtYW1zLnNjdy5jbG91ZDBMBgNVHSAERTBDMAgGBmeBDAECATA3
BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNy
eXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB3AEHIyrHfIkZKEMahOglC
h15OMYsbA+vrS8do8JBilgb2AAABfWwuqRAAAAQDAEgwRgIhAJDHNR59SBFvRIm+
eQkDvMCONckNweCN8nFlHxXHIzaVAiEAyKIGi6qSXlUj0DD6Il1Y99CvqU9UkYo/
1n9c1ntj8YIAdgBGpVXrdfqRIDC1oolp9PN9ESxBdL79SbiFq/L8cP5tRwAAAX1s
LqkyAAAEAwBHMEUCIQDCkL3fleIo7vQ8rT+y9qNsyUrvecBVtbJ0nqDrAR6EEQIg
ciOzQ89wJcdIxAjU6x7KekbHrwbQdrB6gZrPPVwy39kwDQYJKoZIhvcNAQELBQAD
ggEBAKX00v07+hF3XFHoCt/eqxsALWZsAWpOMCVUCwkwAXtmLCz200Ksfin0YqjG
eddzihDjV+V8RBaPKminLW/oOltH337SGEotsX3aGQzEB6h3VxiSZPCLPbNJIB7O
0iGNpahD8qr39JRG1Q6oUrbJ1i5aFzTTGC6k9cl7MGtUn4f2PYhIpZ3f3tqkHZHu
kMMcetRTkMnwX4ac/M+Ah6u/fBXRLC9dvqVgo7zSoW68X0GAAXB0m7HqhQV5mQTF
c3mhEHVeq4ocZO8E4uTkRvZg0Ld5rUeIgFaUsQLe3Odk/mLcQ03vXJyWTf9MN22s
LkP/U8OPEvG7iEdo6bw1oDyl3fY=
-----END CERTIFICATE-----
subject=CN = s3.nl-ams.scw.cloud
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4668 bytes and written 402 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 0C9B35BA8E617CD56543D5FAC8F2A2670EDE5436477FB9C285B71379DF2D2431
Session-ID-ctx:
Resumption PSK: ABCCAF05A71A02EB910D2F159358AB357F11263681D0598D9DC02BF1D9CC2433A01E31213872A5F2FFB6C40DEB57C920
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 43200 (seconds)
TLS session ticket:
0000 - de 5e 8d af 98 04 cc a8-c5 1b c1 82 48 7a e5 ca .^..........Hz..
0010 - a5 46 47 15 22 c2 dc 37-c8 66 3f a1 50 eb 8d 19 .FG."..7.f?.P...
Start Time: 1643302018
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: F02891D974338CB7103F9EF3B0BF68E0672C58AE2F6A0CDF922DB237F15C65BF
Session-ID-ctx:
Resumption PSK: A694ACBA23E4F9C9D613571EE217E1ED5F4C8D60E3066BB0D38F3F8C698EC37A37A38623D8FA4B75C846CC5C619D197B
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 43200 (seconds)
TLS session ticket:
0000 - 59 ce 42 18 06 60 de 84-b6 12 80 5e 94 05 a7 9d Y.B..`.....^....
0010 - 20 50 8e ce db 81 e0 d7-69 a0 8e 59 af d6 7c 9c P......i..Y..|.
Start Time: 1643302018
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: application/xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>RequestTimeout</Code>
<Message>Client request timeout</Message>
</Error>
closedThat was it! I manually removed the X3 root certificate I still had on my system, run /usr/sbin/update-ca-certificates and it now works fine. Thanks for the help.
toupeira
commented
2 years ago Yay, glad you got it sorted out too! :grinning:
kwilczynski
commented
2 years ago Hi @toupeira and @xperseguers,
Sorry for troubles and for late response!
Also, THANK YOU, (both of you) for getting to the bottom of this!
A bit of a backstory:
I moved the archive files to Scaleway as their S3 compatible storage option is quite affordable. I used to host files mirrored on the actual AWS' S3, however, after getting a substantial bill once after a major release of GitLab, I had to reconsider my life choices and look for cheaper option, so to speak.
We are also using my own mirror, as any other mirrors we've tried before eventually ended up blocking our ability reliably download the source code of libmagic either due to heavy throttling or other such limits - this resulted in an influx of build failures, and thus I had to change the strategy a little bit.
Krzysztof
stanhu
commented
2 years ago @kwilczynski I think we just need to get the bundled .tar.gz working. Do you know how we can release a version with this?
Then we can look at moving the mirror to Cloudflare R2 so that it will be free.
kwilczynski
commented
2 years ago Hi @stanhu,
@kwilczynski I think we just need to get the bundled
.tar.gzworking. Do you know how we can release a version with this?
Released 0.5.4. To release it, I simply did bundle exec rake package, and the correct Gem file was created. The only thing I had to do, was to remove content of the pkg sub-directory, otherwise if there was a matching version inside already, then it would stubbornly refuse to build it correctly (the directory containing an entire tree was correct, however, the Gem file wasn't, not sure why).
Then we can look at moving the mirror to Cloudflare R2 so that it will be free.
I have Cloudflare account, but I don't have R2 access. Was it made publicly available already? As in, did they reached a GA status with it? I remember it was invite-only, and such.
Krzysztof
stanhu
commented
2 years ago @kwilczynski Thanks! I think we should document that bundle exec rake package works because I tried bundle exec rake gem and bundle exec rake build, neither of which worked. 😄
Upgrade to 0.5.4 is here: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79388
I have Cloudflare account, but I don't have R2 access. Was it made publicly available already? As in, did they reached a GA status with it? I remember it was invite-only, and such.
Yeah, I think there's a waitlist: https://www.cloudflare.com/r2-storage/
stanhu
commented
2 years ago @kwilczynski We just tagged a GitLab release using ruby-magic v0.5.4. I'm curious if the bandwidth charts drop off like a rock now. 😄
kwilczynski
commented
2 years ago Hi @stanhu,
Very nice!
The bandwidth usage did, indeed, go down as per:
Well done! Thank you!
Krzysztof
The build currently fails with this error:
Apparently it's trying to download https://ruby-magic.s3.nl-ams.scw.cloud/file-5.39.tar.gz, which returns an SSL
PR_END_OF_FILE_ERRORerror in Firefox (same on the root domain https://ruby-magic.s3.nl-ams.scw.cloud/).Is this a temporary issue with S3, or maybe some kind of configuration error?
As a workaround I was able to build the gem using system libraries (after installing
libmagic-devon Debian):