SSL certificate has expired

[vanboom]

We are using ipinfo_io for reverse geocoding. Today we are receiving an error

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired))

The SSL cert for our application is current and valid. But it seems somewhere in the chain the app is attempting to validate against an old expired certificate.

ipinfo_io calls using curl from the command line work fine:

curl https://ipinfo.io/8.8.8.8/city
Mountain View

But the error occurs when reverse_geocode is called from within the app. Any advice would be greatly appreciated. Thanks for a great gem!

[emilianodellacasa]

I can confirm the problem on my application too

[willtcarey]

In our application it doesn't actually seem to be happening with ipinfo.io. It looks like it's the default street address lookup, nominatim. Here's a chunk of code which will reproduce the issue.

uri = URI.parse("https://nominatim.openstreetmap.org")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.get(uri.request_uri)

[willtcarey]

The issue that we've narrowed this down to is our server's ca-certificates package being out of date. Running an update on that package makes the geocoder calls successful. (Our server is CentOS so we use yum but there should be similar packages for Alpine or Debian based distros)

yum update ca-certificates
bin/rails c
Model.first.geocode # This call is now successful.

[vanboom]

@wiltcarey - thank you! Our server is Ubuntu 16.04 and may be affected by this: https://ubuntu.com/security/notices/USN-5089-1

Still researching - but with the older OS version, a simple sudo update-ca-certficates does not resolve the issue.

Our 20.04 servers do not have this issue.

[vanboom]

To fix on Ubuntu 16.04, related to https://ubuntu.com/security/notices/USN-5089-1

1. Edit /etc/ca-certificats.conf
2. Comment out the line pertaining to DST_Root_CA_X3
3. Run update-ca-certificates Note that the offending cert has been removed...

0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

Many thanks to @willtcarey for pointing us in the cert direction!

[alexreisner]

Thanks @vanboom and @willtcarey for posting your solutions!

[Nowaker]

For anyone stuck on archaic systems:

OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

[ilasno]

@alexreisner Thanks so much for your work on this, and your quick responses!

Wondering about error handling on this.. i looked at https://github.com/alexreisner/geocoder#error-handling and it says "By default Geocoder will rescue any excepotionsraised by calls to a geocoding service and return an empty array".

But it appears that this SSL certificate issue does not fall into that rescuing? We use: geocoded_by :address_for_geocoding_and_mapping after_validation :geocode, :if => in a model, and this certificate issue just blew up the entire model.save call: /webdocs/rvm/rubies/ruby-2.5.8/lib/ruby/2.5.0/net/protocol.rb:44:in 'connect_nonblock' /webdocs/rvm/rubies/ruby-2.5.8/lib/ruby/2.5.0/net/protocol.rb:44:in 'ssl_socket_connect' /webdocs/rvm/rubies/ruby-2.5.8/lib/ruby/2.5.0/net/http.rb:985:in 'connect' /webdocs/rvm/rubies/ruby-2.5.8/lib/ruby/2.5.0/net/http.rb:920:in 'do_start' /webdocs/rvm/rubies/ruby-2.5.8/lib/ruby/2.5.0/net/http.rb:909:in 'start' /webdocs/rvm/rubies/ruby-2.5.8/lib/ruby/2.5.0/net/http.rb:609:in 'start' geocoder (1.6.5) lib/geocoder/lookups/base.rb:299:in 'make_api_request' geocoder (1.6.5) lib/geocoder/lookups/base.rb:249:in 'fetch_raw_data' geocoder (1.6.5) lib/geocoder/lookups/base.rb:195:in 'fetch_data' geocoder (1.6.5) lib/geocoder/lookups/nominatim.rb:36:in 'results' geocoder (1.6.5) lib/geocoder/lookups/base.rb:46:in 'search' geocoder (1.6.5) lib/geocoder/query.rb:11:in 'execute' geocoder (1.6.5) lib/geocoder.rb:22:in 'search' geocoder (1.6.5) lib/geocoder/stores/base.rb:100:in 'do_lookup' geocoder (1.6.5) lib/geocoder/stores/active_record.rb:298:in 'geocode'

Is there a safeguard we can employ that would allow the save, in a case like this, just without geocode values?

[alexreisner]

@ilasno Good question. This exception isn't caught because I haven't encountered it before! I'd be open to a pull request that rescued from this exception and printed some useful advice (like what's been posted above).

[Nowaker]

@alexreisner I would really advise against doing that. How would I know if my geocoder fails, if it's swallowing the exception? The decision to ignore the exception should always be explicit or configurable but default off.

[alexreisner]

@Nowaker I think I agree with you but let me explain my thinking on this because there are a few things to consider, and other opinions would be helpful.

Geocoder currently rescues from other connection errors like SocketError and Errno::ECONNREFUSED, and issues a warning (unless the user has configured Geocoder to re-raise them). The reasoning is that these errors are likely caused by temporary issues with a network or API, not resolvable by a developer, so silent (or nearly silent) failure is the best default for most apps.

An OpenSSL::SSL::SSLError is a bit different. The part I'm not sure about is whether it always indicates action required by a developer. If so, the best course of action is to make no change to Geocoder. (And in that case, @ilasno, your solution is to enclose your geocoding in a rescue block.) Even if not, the fact that it sometimes indicates action required by a developer probably means we shouldn't ever rescue from it.

Does that all sound right?

[Nowaker]

OpenSSL::SSL::SSLError means Geocoder is running in a very old environment that is missing the latest CA certificates OR is running an outdated version of OpenSSL. One or the other will trigger the error. Connecting to any HTTPS endpoint with a cert issued by Let's Encrypt will trigger the same error. Any HTTP client is affected equally - not just Geocoder. A possible workaround for that is OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE in initializers.

Based on the above, I believe this is not a Geocoder issue. But we can sure offer some accommodations to workaround the issue but they should be optional (default off). Along with, if verify is enabled (default), it should still rescue a potential OpenSSL::SSL::SSLError, and raise a Geocoder exception with a friendly error like TLS verification error. This may be caused by old CA certs or outdated OpenSSL version. Please upgrade your system or workaround by disabling TLS verify with Geocoder.configure(verify: OpenSSL::SSL::VERIFY_NONE).

[alexreisner]

Thanks @Nowaker! That's extremely helpful, and I really like your suggestion. However, as you say, it's not a Geocoder issue and it seems like the best place for the workaround would be in an app-level initializer (especially if Geocoder isn't the only thing using OpenSSL).

So I think I'm in favor of rescuing and re-raising an error with a helpful message, but not in favor of adding the config option.

But I'm open to discussion.

[ilasno]

Wow, really appreciate the quick and thoughtful responses on this @Nowaker @alexreisner . My $.02:

While probably rare, isn't it true that OpenSSL::SSL::SSLErrorcould also occur if the geocoding provider's certificate has expired? If so, this would still not be a Geocoder issue, but it may then fall into the 'not resolvable by a developer' category. Admittedly, 'it sometimes indicates action required by a developer' remains true, though.

As a developer, i'd agree with 'rescuing and re-raising an error with a helpful message, but not in favor of adding the config option'.

That just leaves my (perhaps newbish) question: How would i 'enclose your geocoding in a rescue block' if the geocoding is triggered by: geocoded_by :address_for_geocoding in my model? I don't see an explicit call to 'geocode' that i can wrap in a rescue block.

[alexreisner]

Thanks @ilasno. Regardless of when it's currently raised, SSLError is such a non-specific name that we can't be assured it won't be raised elsewhere in the future. We'll need to word the helpful message carefully, and not be too confident that we know what the developer receiving the error should do.

To enclose geocoding in a rescue block, leave your geocoded_by line as is. You probably also have a line after_validation :geocode. If so, you can define a method like this:

def geocode
  super
rescue
end

I think that will do what you want.

[ilasno]

Ahh, right, thanks so much @alexreisner !

[dylanbromby]

Not to split hairs, but @vanboom states you should comment the cert reference but you actually need to deselect it.

This:

!mozilla/DST_Root_CA_X3.crt

Not this:

#mozilla/DST_Root_CA_X3.crt

So:

1. vi /etc/ca-certificates.conf
2. Deselect DST_Root_CA_X3 (see above)
3. update-ca-certificates

This fixed the issue for us on Ubuntu 16.04. And for those running Docker, remember to update /etc/ca-certificates.conf in your container vs. your host 😉 .

[vanboom]

@dylanbromby you are correct - apologies for the typo ;)

[basicfeatures]

Unfortunately having the same problem: https://gist.github.com/basicfeatures/90c861bf9e8ee89f888d439f4cb9cbfe - any clues?

[dylanbromby]

Unfortunately having the same problem: https://gist.github.com/basicfeatures/90c861bf9e8ee89f888d439f4cb9cbfe - any clues?

Have you tried the steps in my post above?

[MrGurns]

@basicfeatures You might need to restart your passenger/apache/nginx. Worked for me once i restarted passenger.

[basicfeatures]

@dylanbromby @MrGurns Hi! I run OpenBSD which has its own rewrite of OpenSSL (LibreSSL). Turned out I was running a slightly older release of OpenBSD and a simple upgrade took care of the problem 👍 Thanks guys!