SSL verification, Windows in particular (was: Soften HTTPS; was: Geotoad fails when starting)

[FriedrichFroebel]

Error message (first part does not seem to be problem as it appeared before, too).

I call the script via command line: C:\Geocaching\Geotoad>geotoad.rb

Then this output is printed to the command line. After this the script ends running.

C:/Geocaching/Geotoad/lib/output.rb:37: warning: duplicated key at line 128 ignored: "
AN"
C:/Geocaching/Geotoad/lib/output.rb:48: warning: duplicated key at line 131 ignored: "
IN"
C:/Geocaching/Geotoad/lib/output.rb:129: warning: duplicated key at line 143 ignored:
"ZUM"
C:/Geocaching/Geotoad/lib/output.rb:130: warning: duplicated key at line 144 ignored:
"ZUR"
(===) GeoToad 3.24.1 (Ruby 2.2.3p173/2015-08-18 on i386-mingw32)
 ***  Cannot connect to raw.githubusercontent.com:443: SSL_connect returned=1 errno=0 state=S
SLv3 read server certificate B: certificate verify failed
( - ) Offline mode: not fetching https://raw.githubusercontent.com/wiki/steve8x8/geotoad/Curr
entVersion.md
C:/Geocaching/Geotoad/lib/shadowget.rb:257:in `fetchRemote': undefined method `length'
 for nil:NilClass (NoMethodError)
        from C:/Geocaching/Geotoad/lib/shadowget.rb:169:in `fetch'
        from C:/Geocaching/Geotoad/geotoad.rb:215:in `versionCheck'
        from C:/Geocaching/Geotoad/geotoad.rb:1007:in `<main>'

[steve8x8]

If you change VERIFY_PEER to VERIFY_NONE in lib/shadowget.rb, line 285 (or comment this line out, and uncomment the previous one), does this fix the issue? If so, can you reproduce the issue without this fix, but with other Ruby versions? (Hint: the "pik" gem allows to switch between different versions.)

[FriedrichFroebel]

Yes, if I uncomment line 284 and comment line 285 out, it seems to work.

[steve8x8]

Apparently your Ruby installation isn't able to verify the certificate presented by github. Applying the following mini-patch (lib/shadowget.rb, line 256), and reverting the VERIFY_* change, should enable you to proceed (without checking the release notes, of course):

-    data = fetchURL(@url)
+    data = fetchURL(@url).to_s

There will still be warnings. Can you login to gc.com, even with VERIFY_PEER set? If yes, your set of root certificates is incomplete. (http://notetoself.vrensk.com/2008/09/verified-https-in-ruby/ explains how it works and how it could be fixed, but I won't distribute static pem files when there's a solution at OS level. Blame MS...)

[FriedrichFroebel]

Using the described change in lib/shadowget.rb (https://github.com/steve8x8/geotoad/issues/329#issuecomment-151784184), there is the same output as in #328.

[steve8x8]

There seems to be a common denominator: Windows. Since I'm still stuck with an old XP VM (which wouldn't run Ruby >= 2.1), I cannot properly test. Is there another platform that shows this certificate verification issue? If not, it would be possible to detect a Windows platform and disable verification, thus reversing the hardening that happened as a response to issue #322 and #326 . Not really desirable, IMHO :(

[steve8x8]

I hate it, but there's a 3.24.2-RC1 now tagged (which should also be available as a Github release tarball). I'll certainly put that one under my pillow for another night or two, and try to come up with a real release next week. Feedback is welcome! For now, only Windows platforms get the "verify" switch set to NONE. Is this enough?

[steve8x8]

3.24.2 is out now, incorporating the "softening" code together with a list of warnings. I'm still building Windows Installer packages on an XP VM, with Ruby limited to 2.0 - and I cannot really test the result, so it's possible it's still broken. If so, try to use the tarball, and give me feedback.

[FriedrichFroebel]

I have never used the Windows installer - I have always been working with the tarball. The current version (tarball) works for me without problems.

[IByte]

GeoToad 3.24.2 and the installer work well for me on Windows 10, thanks! SSL certificate verification is suppressed.

[steve8x8]

Still looking for a clean method to use certificate verification within Windows, but apparently no-one would ever use system-wide certificate storage (which doesn't exist in turn, chicken--egg?). This is a kludge, and probably will be forever, but at least it's now clearly marked as such. Is there a single Unix variant around that has SSL cert problems like this that cannot be resolved in a general way?

[IByte]

I'm not sure about Unixes, but I use GeoToad as a component of a tool I wrote in PHP that expands a few fields of the GPX files (e.g. automatic county lookup), which runs on Windows. To log in to and fetch pages from geocaching.com, I use the cURL library (see http://curl.haxx.se/), to which I supplied a file with root certificates. It looks like Ruby bindings for libcurl are available, it might be of interest to you to look into it (I don't know Ruby myself).

[steve8x8]

For Unices, certificate magic probably isn't necessary. How would I extract the certificate chain(s) required for authentication of www.geocaching.com and raw.githubusercontent.com? I presume this could be done with openssl but never had to do this. (maps.google.com doesn't speak https with us - yet.)

[IByte]

Here are a few references on how to do it. You can download a pre-compiled file of certificates from the cURL site or extract them from Firefox. I tried attaching the file I use, but Github won't let me.

http://stackoverflow.com/questions/6400300/https-and-ssl3-get-server-certificatecertificate-verify-failed-ca-is-ok

http://curl.haxx.se/docs/caextract.html

[steve8x8]

Thanks. Could someone with a non-installer Windows version try the following:

• download http://curl.haxx.se/ca/cacert.pem
• in geotoad.rb (3.24.2), line 1014 disable the check for Windows (comment out 4 lines if...end) - this will cause the problem to reappear
• in a cmd window, "set SSL_CERT_FILE=c:\whateverpath\cacert.pem", then run "ruby geotoad.rb" as usual - can you still reproduce the problem?

A most general solution would check for the existence of this environment variable (and the file), and switch on SSL verification even for windows, with the fallback to unverified connections. If the above works I'd settle for this, and eventually close this issue. (A cert file will become part of the GeoToad distribution.)

Well, here's the central part of the patch (line numbers may be shifted a bit as this is against the current version in trunk):

--- a/geotoad.rb    2015-11-11 12:20:11.000000000 +0100
+++ b/geotoad.rb    2015-11-14 13:59:53.109959253 +0100
@@ -1023,7 +1023,10 @@
 # work around (only?) Windows not being able to verify peer
 # http://stackoverflow.com/questions/170956/how-can-i-find-which-operating-system-my-ruby-program-is-running-on
 # better use RbConfig::CONFIG['host_os']?
-if RUBY_PLATFORM.downcase =~ /djgpp|(cyg|ms|bcc)win|mingw|wince|emx/
+if ENV['SSL_CERT_FILE'] and File.readable?(ENV['SSL_CERT_FILE'])
+  displayInfo "HTTPS will use SSL cert file #{ENV['SSL_CERT_FILE']}"
+  #$SSLVERIFYMODE = OpenSSL::SSL::VERIFY_PEER
+elsif RUBY_PLATFORM.downcase =~ /djgpp|(cyg|ms|bcc)win|mingw|wince|emx/
   displayWarning "HTTPS will not verify peer identity!"
   $SSLVERIFYMODE = OpenSSL::SSL::VERIFY_NONE
 end

Feedback is appreciated!

[steve8x8]

Anintermediate build 3.24.2+20151115 is available from the work-in-progress download area now, This will become the basis for the first 3.25.0 release candidate.

[steve8x8]

The 3.25.0 Windows Installer comes with a CA root-certificate collection, and it tries to install that. I have no means to test this, and discourage using the Installer completely. If you choose to go for a recent ruby-installer and the tarball instead, please check the manual page for how to make the root-certs available.

[steve8x8]

After almost 5 months without an addition, it's time to close this issue.