search

nhosoya / omniauth-apple

OmniAuth strategy for Sign In with Apple
MIT License
264 stars 101 forks source link

Provide User-Agent when fetching JWKs #59

Closed mikker closed 3 years ago

mikker commented 4 years ago

I was receiving 403s without the User-Agent

erkie commented 3 years ago

Seems like Apples servers are blocking certain IP's without user agents. (We use DigitalOcean). Amazing that you found this.

mikker commented 3 years ago

This is still a problem when using DigitalOcean. Anything I can do to get this merged?

Meat-Chopper commented 3 years ago

In fact, when you don't specify a UA, it's not missing. By default, Net::HTTP sets "user-agent" => "Ruby", so in this commit you just replace 'Ruby' with 'ruby/omniauth-apple'

mikker commented 3 years ago

Somehow it fixes a problem that some of us had, sooo 🙃

btalbot commented 3 years ago

From other reports, this "missing ua" issue seems to be restricted to some Digital Ocean clients.

It seems that

  1. something in your stack is configuring the Net::HTTP library and clearing the default UA, or
    • maybe a custom DO Ruby build?
  2. some infrastructure at DO must be stripping out the "Ruby" ua value (a transparent mitm forward proxy?), or
    • Is there a DO CA in your TLS trust database?
  3. some DO traffic is being routed to special apple servers that are treating the "Ruby" ua value as missing.
erkie commented 3 years ago

All three are incorrect. Apple has blocked DigitalOcean IP's from accessing this service with a user-agent of "Ruby".

user@do-server$ curl -vvv "https://appleid.apple.com/auth/keys" -H"User-agent: Ruby"
*   Trying 17.32.194.37...
* TCP_NODELAY set
* Connected to appleid.apple.com (17.32.194.37) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=California; serialNumber=C0806592; C=US; ST=California; L=Cupertino; O=Apple Inc.; OU=IdMS Operations; CN=appleid.apple.com
*  start date: Jun  9 00:00:00 2020 GMT
*  expire date: Jun  9 12:00:00 2021 GMT
*  subjectAltName: host "appleid.apple.com" matched cert's "appleid.apple.com"
*  issuer: C=US; O=DigiCert, Inc.; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA-3
*  SSL certificate verify ok.
> GET /auth/keys HTTP/1.1
> Host: appleid.apple.com
> Accept: */*
> User-agent: Ruby
>
< HTTP/1.1 403 Forbidden
< Server: Apple
< Date: Mon, 17 May 2021 20:23:47 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
<
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Apple</center>
</body>
</html>

vs

user@do-server$ curl -vvv "https://appleid.apple.com/auth/keys" -H"User-agent: something-else"
*   Trying 17.32.194.37...
* TCP_NODELAY set
* Connected to appleid.apple.com (17.32.194.37) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=California; serialNumber=C0806592; C=US; ST=California; L=Cupertino; O=Apple Inc.; OU=IdMS Operations; CN=appleid.apple.com
*  start date: Jun  9 00:00:00 2020 GMT
*  expire date: Jun  9 12:00:00 2021 GMT
*  subjectAltName: host "appleid.apple.com" matched cert's "appleid.apple.com"
*  issuer: C=US; O=DigiCert, Inc.; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA-3
*  SSL certificate verify ok.
> GET /auth/keys HTTP/1.1
> Host: appleid.apple.com
> Accept: */*
> User-agent: something-else
>
< HTTP/1.1 200
< Server: Apple
< Date: Mon, 17 May 2021 20:24:27 GMT
< Content-Type: application/json;charset=ISO-8859-1
< Content-Length: 1439
< Connection: keep-alive
< Cache-Control: no-store
< Pragma: no-cache
< Host: appleid.apple.com
<
{
  "keys": [
    {
      "kty": "RSA",
      "kid": "86D88Kf",
... etc

My local laptop:

me@local-laptop$ curl -vvv "https://appleid.apple.com/auth/keys" -H"User-agent: Ruby"

*   Trying 17.111.105.242...
* TCP_NODELAY set
* Connected to appleid.apple.com (17.111.105.242) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionCountryName=US; jurisdictionStateOrProvinceName=California; serialNumber=C0806592; C=US; ST=California; L=Cupertino; O=Apple Inc.; OU=IdMS Operations; CN=appleid.apple.com
*  start date: Jun  9 00:00:00 2020 GMT
*  expire date: Jun  9 12:00:00 2021 GMT
*  subjectAltName: host "appleid.apple.com" matched cert's "appleid.apple.com"
*  issuer: C=US; O=DigiCert, Inc.; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA-3
*  SSL certificate verify ok.
> GET /auth/keys HTTP/1.1
> Host: appleid.apple.com
> Accept: */*
> User-agent: Ruby
>
< HTTP/1.1 200
< Server: Apple
< Date: Mon, 17 May 2021 20:25:01 GMT
< Content-Type: application/json;charset=ISO-8859-1
< Content-Length: 1439
< Connection: keep-alive
< Cache-Control: no-store
< Pragma: no-cache
< Host: appleid.apple.com
<
{
  "keys": [
    {
      "kty": "RSA",
      "kid": "86D88Kf",
      "use": "sig",
mikker commented 3 years ago

Probably someone has been doing something malicious from DO IPs with Ruby and so they've just banned that combo. If you're hesitant to change the UA for everyone, I could turn it into an option instead?

Meat-Chopper commented 3 years ago

Would it be reasonable to write to Apple support?

erkie commented 3 years ago

It makes a lot of sense to to set a custom User-agent for this application. It's just good practice for a requester to identify who itself is. I don't see why not get this merged.

I would also assume if we could get in touch with Apple tech support somehow they would say "identify yourself with the user-agent"...

Meat-Chopper commented 3 years ago

Since you have already identified yourself with the "Ruby" UA, they may ban "ruby/omniauth-apple" as well. It's better to have an option to customize the UA.

erkie commented 3 years ago

They may, but then they know who to contact if a client is misbehaving. At this point we should just venture to get a fix merged. @nhosoya what would it take to get this fix merged? what do we need to add/remove to this PR?

nhosoya commented 3 years ago

I am sorry, but I have not been able to follow the issues and discussions. Please give me some time.

nhosoya commented 3 years ago

I've caught up with everyone.

@erkie Thanks for everything.

I want to make sure that when you request from DigitalOcean IP with a UserAgent of "ruby/omniauth-apple", you get a normal response. (Please post the logs)

Since this verification is for the case where the UserAgent is "something-else", the results of this PR change is not a guaranteed result.

Once this has been verified, I will merge this PR.

erkie commented 3 years ago

Thanks @nhosoya for looking into this. Great idea to verify that the PR User-agent works :) Below is the verification.

do-server:~$ curl -vvv "https://appleid.apple.com/auth/keys" -H"User-agent: ruby/omniauth-apple"
*   Trying 17.32.194.6...
* TCP_NODELAY set
* Connected to appleid.apple.com (17.32.194.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=California; serialNumber=C0806592; C=US; ST=California; L=Cupertino; O=Apple Inc.; OU=IdMS Operations; CN=appleid.apple.com
*  start date: Jun  9 00:00:00 2020 GMT
*  expire date: Jun  9 12:00:00 2021 GMT
*  subjectAltName: host "appleid.apple.com" matched cert's "appleid.apple.com"
*  issuer: C=US; O=DigiCert, Inc.; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA-3
*  SSL certificate verify ok.
> GET /auth/keys HTTP/1.1
> Host: appleid.apple.com
> Accept: */*
> User-agent: ruby/omniauth-apple
>
< HTTP/1.1 200
< Server: Apple
< Date: Wed, 19 May 2021 10:48:56 GMT
< Content-Type: application/json;charset=ISO-8859-1
< Content-Length: 1439
< Connection: keep-alive
< Cache-Control: no-store
< Pragma: no-cache
< Host: appleid.apple.com
<
{
  "keys": [
    {
      "kty": "RSA",

And to verify that Ruby didn't work:

do-server$ curl -vvv "https://appleid.apple.com/auth/keys" -H"User-agent: Ruby"
*   Trying 17.32.194.6...
* TCP_NODELAY set
* Connected to appleid.apple.com (17.32.194.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=California; serialNumber=C0806592; C=US; ST=California; L=Cupertino; O=Apple Inc.; OU=IdMS Operations; CN=appleid.apple.com
*  start date: Jun  9 00:00:00 2020 GMT
*  expire date: Jun  9 12:00:00 2021 GMT
*  subjectAltName: host "appleid.apple.com" matched cert's "appleid.apple.com"
*  issuer: C=US; O=DigiCert, Inc.; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA-3
*  SSL certificate verify ok.
> GET /auth/keys HTTP/1.1
> Host: appleid.apple.com
> Accept: */*
> User-agent: Ruby
>
< HTTP/1.1 403 Forbidden
< Server: Apple
< Date: Wed, 19 May 2021 10:52:17 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
<
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Apple</center>
</body>
</html

Thanks for taking the time!

nhosoya commented 3 years ago

@erkie Thanks a lot!!!

There is an idea to make UserAgent optionally configurable, but we think that a fixed UserAgent will be enough for now.

So, I will merge and release a new version.

erkie commented 3 years ago

Thank you! And thanks for this library. It's excellent :)

mikker commented 3 years ago

💙💚💛💜❤️