What's the right way to use TLS for IOS APP?

[amazingyyc]

What's the right way to use TLS for IOS APP?

I use swift-grpc as my ios APP network api, but when enable TLS for backend API the network request will fail! So I want to know the right way to enabel TLS for backend and ios APP.

My method and What I have tried.

My backend system structure: 1 nginx machine with 2 APP service, nginx config like below: without TLS config:

upstream grpcservers {
        server ip0:50051;
        server ip1:50051;
}

server {
        listen 80 http2;
        location / {
                grpc_pass grpc://grpcservers;
        }
}

with TLS config:

// use below command to generate crt/key file.
// openssl req -newkey rsa:2048 -nodes -keyout server.key -x509 -days 3650 -out server.crt

upstream grpcservers {
        server ip0:50051;
        server ip1:50051;
}

server {
        listen 443 ssl http2;
        ssl_certificate ssl/server.crt;
        ssl_certificate_key ssl/server.key;
        location / {
                grpc_pass grpc://grpcservers;
        }
}

The gprc service start like:

server.add_insecure_port(f'[::]:{conf.grpc_port}')

Below is How I config the ios APP client (ref:https://github.com/grpc/grpc-swift/blob/main/Examples/Google/SpeechToText/Sources/SpeechService.swift)

let group = PlatformSupport.makeEventLoopGroup(loopCount: numberOfThreads)

// Setup a logger for debugging.
var logger = Logger(label: "gRPC", factory: StreamLogHandler.standardOutput(label:))
logger.logLevel = .debug

let channel = ClientConnection
      .secure(group: group)
      .withTLS(certificateVerification: .none)
      .withBackgroundActivityLogger(logger)
      .connect(host: host, port: port)

var callOptions = CallOptions.init(logger: logger)

client = Client.init(channel: channel, defaultCallOptions: callOptions)

What my question is:

• without TLS config I can request backend success with nginx ip:port and host name:port.
• with TLS config and add .withTLS(certificateVerification: .none), I can request backend success with nginx ip:port but failed with host name:port.
• with TLS config and remove .withTLS(certificateVerification: .none), both of nginx ip:port and host name:port will be failed.

So I'm not good at the network encryption, So any can help to check What I did wrong? and I can fix it.

[williamMillington]

Hello!

I recently set up gRPC on my iOS app using TLS –and had some difficulty– so maybe I can help.

• Unfortunately some of the examples –such as the Google Speech service– have not been updated in a while. ClientConnection.secure(group: group) is now deprecated, so I believe you will need to use: let channel = ClientConnection.usingPlatformAppropriateTLS(for: group)

• It is also a good idea to change PlatformSupport.makeEventLoopGroup(loopCount: numberOfThreads) to include networkPreference: .best (PlatformSupport.makeEventLoopGroup(loopCount: numberOfThreads, networkPreference: .best). It will still work if you do not, but my understanding is that it will help gRPC-swift choose the right background components automatically.

• Delete .withTLS(certificateVerification: .none). The reason your connection works when you include it is because it turns off TLS. This can sometimes be useful, but it is not what you're looking for (I think you probably suspected that anyway).

• You will need to changeserver.add_insecure_port(f'[::]:{conf.grpc_port}') because it creates a server port without TLS. Use your server certificate and key to create credentials: credentials = grpc.ssl_server_credentials( [server_certificate, server_key] ) and then use those to create a secure port:
  server.add_secure_port('[::]:{conf.grpc_port}', credentials)

It looked like you're using python on the server side, which I'm not very familiar with. Because of that, the above lines of code are not completely correct, so you will need to find out the specifics of what you want your configuration to be. I found this guide https://realpython.com/python-microservices-grpc/ to be helpful. There is a lot of information there that is not related to what you are doing, but there is much information that is, and the author explains what they are doing at every single step.

• I believe that the reason nginx ip:port works, and hostname:port does not, is that when you created your server with server.add_insecure_port(f'[::]:{conf.grpc_port}') you did not give it a hostname, only a port, so by default it is only responding to the ip address. I could be wrong, but when you create a gRPC server I believe that the string'[::]' simply means 'default ip address' (but I am not 100% sure about this).

Finally:

• Unfortunately, I do not think that changing those lines of code will fully solve the problem. Generally speaking, the process you're using to create your server certificate is incomplete. openssl req -newkey rsa:2048 -nodes -keyout server.key -x509 -days 3650 -out server.crt will generate a certificate and a key, but (usually) this is not enough, because any server certificate will need to be signed by a trusted Certificate Authority in order for anyone to trust it (including yourself). There are certain times when this is not needed, but you will have to put together a bit of a custom solution.

I think I could provide some help about where to start (I had to do this for my own application) but I don't think it would be appropriate here. This can be a very big topic depending on what you want to do, and is not strictly related to gRPC-swift. I think it would be better to recreate the question on StackOverflow and post the link here; I'd be happy to go into more detail.

I hope some/any of that is helpful!

[amazingyyc]

@williamMillington I'm really really thank you for you spend time to help me. Maybe I'm not explain my backend system architecture clear. My backend system is below: The ios APP connect nginx use TLS connection (use swift-grpc), the nginx connect backend GRPC service (write by python) use insecure GRPC, (nginx connect with python GRPC service use insecure channel, because the python service running in internal network, so I think it's not necessary use secure channel).

                                    |-----(insecure connect)---GRPC (python)
                                    |  
ios APP ---(TLS connect)-------nginx
                                    |
                                    |-------(insecure connect)-----GRPC (python)

What my problem is:

• When disable TLS between ios APP and nginx. everything works well. My app works well can call nginx success with ip address and hostname.
• When enable TLS between ios APP and nginx, I only can connect nginx with ip address (with .withTLS(certificateVerification: .none)), when use hostname it will fail.

Firstly thanks you for your support the new API.

• About the .withTLS(certificateVerification: .none) I have another opinion. From the swift-grpc code comment: Whether to verify remote certificates. Defaults to .fullVerification if not otherwise configured. So I think when connect with: .withTLS(certificateVerification: .none) means not check the TLS server certificate, just use it , So That's why I can connect success with ip address (I'm not good at network encryption, it's just my gauss.)
• About the server.add_insecure_port(f'[::]:{conf.grpc_port}') you are right I use python as my backend API. I try to change the [::] to 0.0.0.0, looks like it not works. From the grpc-python API document: [::] means bind every ip address on current machine, So I think it's not cause.
• About generate the crt/key. I generate it by myself because I think my backend nginx is only serving for my own app not expose a public API for other app. So I use my own crt/key and close the certificate check in my own app (like use:.withTLS(certificateVerification: .none)) than the connection will be righ. Actually I try to use a "trusted Certificate Authority" 's crt/key, it still fail with hostname (Maybe I lose some thing, I will try it agian)

[williamMillington]

You're welcome!

Thank you for clarifying, I think I see now. I'd never heard of nginx before and I didn't understand the role it was playing,

• I see how you're right: gRPC should not need a secure port.
• I also think you're right to use '[::]' (although it might be worth trying out adding the port with 'hostname' instead. I've personally had difficulty getting gRPC to work with hostnames, and gave up and use IP addresses only. But I'm also connecting to a server I don't control who insists on addresses. You might have more success since you can control both sides)
• I think you're completely correct about not needing a trusted authority, it will just involve a few more steps to set up. My own app is like this as well. I only brought it up because I wasn't sure how much you knew about creating them (I myself knew nothing when I started my app).

So I think when connect with: .withTLS(certificateVerification: .none) means not check the TLS server certificate, just use it

To clarify: Do you mean "Use it" as in "The server will know that it's talking to my app, because my app will be using the server's certificate to encrypt its gRPC calls"? If that is the case, then I believe this is where the misunderstanding is occurring.

• When you use the line let client = ClientConnection.usingPlatformAppropriateTLS(for: group) you are telling gRPC-swift: "I am creating a connection to a server that uses TLS to prove who it is". If you provide no further TLS configuration, then gRPC will attempt to verify the certificate that the server provides by asking iOS for a list of Certificate Authorities that iOS trusts, and checking if the certificate was signed by any of them. This is why I believe the attempts to use your own certificate have failed: it was not signed by a Certificate Authority that is trusted by iOS, so gRPC-swift rejected it. There are also many reasons why the connection might have failed despite a crt/key from a trusted Authority; I can't say without more details.
• When you use the line .withTLS(certificateVerification: .none) you are telling gRPC-swift: "Don't verify the server's certificate, assume it's legit". This is important because your current setup only involves the client verifying the server, not the server verifying the client (More on this later). This means someone could insert themselves into your connection right at the beginning, capture the server's certificate, and send you their own (Man-In-The-Middle attack). You will never detect this if you turn off certificate verification.

Now, you can't turn off TLS, because you want to trust the server, but gRPC-swift doesn't trust a certificate that hasn't been signed by someone in its chain of trust, so what do you do? Insert yourself into the chain of trust!

• Take your server's certificate (in .pem format. I believe its literally the same thing as .crt, you can just change the extension. That's what I've been doing.) and include it in your app somewhere

• Load it into an NIOSSLCertificate object:

   var serverCertificate: NIOSSLCertificate?
   do { 
          let filepath = Bundle.main.path(forResource: "serverCertificate",
                                      ofType: "pem")!
          let certURL = URL(fileURLWithPath: filepath)
          let certBytes = try Data(contentsOf: certURL)
          let CA_chain = try NIOSSLCertificate.fromPEMBytes(Array(certBytes))
  
          // NIOSSLCertificate.fromPEMBytes will return an array of all the certificates in
          // the file, but there should only be one in yours so you can just immediately pull it out
          serverCertificate = CA_chain[0] 
      } catch {
          print(DEBUG_TAG+"Error loading certificate from file \(error)")
      }

• and add .withTLS(trustRoots: .certificates([serverCertificate!]) ) when you create your ClientConnection. This will tell gRPC: "Here is the list of trusted certificates I want you to compare the server's certificate to". The verification should now work, because the server's certificate is being verified against itself.
• Also, I'm just finding this now, this might be useful for connecting to "hostname:port" (and I'm going to try it in my own app to see if it will help with my own problems around that): .withTLS(serverHostnameOverride: String?). According to the docs:

  Sets a server hostname override to be used for the TLS Server Name Indication (SNI) extension. The hostname from connect(host:port) is for TLS SNI if this value is not set and hostname verification is enabled.

  (Possibly requires some sort of 'usehostnameVerification' setting on the server to be activated. I believe it's an available setting for gRPC but I don't know about NginX) But if you find you still can't get hostname to work then you may need to add the IP address of the server into it's certificate as a "Subject Alternate Name" extension, which is what I've had to do. Here's the script I've been using to generate my certificates:

  
  CN_SERVER=YourHostname
  IP_ADDRESS="You.rSe.rve.rIP"

EXT_SubAltName="subjectAltName=IP:${IP_ADDRESS}"

says what we expect to use this certificate for

EXT_ExKeyUsage="extendedKeyUsage=serverAuth,clientAuth"

make sure you use openssl instead of LibreSSL, which does not have -addext (which allows you to add extensions)

I'm on a mac, so this is where that is located for me. Might be different for your machine.

ssl=/usr/local/opt/openssl/bin/openssl

$ssl req -newkey rsa:2048 -nodes -keyout rootkey.key -x509 -days 30 -out root.crt -subj "/CN=${CN_SERVER}" -addext $EXT_SubAltName -addext $EXT_ExKeyUsage

All of this will allow your ClientConnection to trust that the server is who it says it is. You get to generate your own certificates, and if someone slips you a different certificate during a MITM, gRPC will still say "ummmmmmmm...you are _not_ on the list, bud". 

There are some more steps involved if you want what is called Mutual TLS –where the server also verifies that the client is who they say they are– but they are very similar to the above steps so I won't get into them.

I found this page to be particularly helpful: [https://dev.to/techschoolguru/load-balancing-grpc-service-with-nginx-3fio#config-nginx-for-grpc-with-tls](url). The author uses Go instead of Python, but otherwise it has a lot of the information I think you're looking for. There are some links at the top to other posts in the same series which might also be good to look through. 
This other person does it without going to all the trouble of creating their own Certificate Authority (and also uses Python) [https://www.sandtable.com/using-ssl-with-grpc-in-python/](url) but NginX might be more picky than plain gRPC.

[Lukasa]

Given that you can connect with IP address, but not with hostname, there are only a couple of possible problems.

1. The DNS is not set up correctly and so the hostname does not resolve to the correct IP address.
2. nginx doesn't know it should be presenting the TLS certificate in question.

Can you run openssl s_client -connect <host>:<port> -servername <host> and print the output here?

[amazingyyc]

Thanks williamMillington and Lukasa for your answer. I try some methods today. It's till cannot call success TLS. But I get some learning.

• About .withTLS(certificateVerification: .none) I think williamMillington you are right. It's not what I thought. It's just close the TLS channel.
• Another thing is I think I misunderstanding the server.crt/server.key, I regenerate the CA file following below steps:

  
  // step 1: generate CA private key and certificate by follow command
  openssl req -newkey rsa:2048 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt

// Than step2: generate certificate for my own domain-name. openssl req -newkey rsa:2048 -nodes -sha256 -keyout domain.key -new -out domain.csr

// step3: sign domain.csr by ca.crt openssl x509 -CA ca.crt -CAkey ca.key -in domain.csr -req -days 365 -out domain.crt -CAcreateserial -sha256

After above step I think I have already sing my own certificate by my self. Than I config it in my nginx server like below:

server { listen 443 ssl http2;

            ssl_certificate ssl/domain.crt;
            ssl_certificate_key ssl/domain.key;

            location / {
                    # The 'grpc://' prefix is optional; unencrypted gRPC is the default
                    grpc_pass grpc://grpcservers;
                    # root   html;
                    # index  index.html index.htm;
            }
    }

I think above step is the right way to generate own certificate

Because I try to request the server on web browser by https, As expected The web browser(chrome and safari) tell me I it's not a valid certificate and block me to visit my own server. I close the waning and accept trust the  certificate, finally I can visit my server with my own generate certificate. 
<img width="565" alt="截屏2021-12-07 01 35 15" src="https://user-images.githubusercontent.com/3955487/144894113-885106fb-8a0f-4f59-b21a-da44eec538c5.png">

- And base on above setup I try add `withTLS(certificateChain:) withTLS(trustRoots:) withTLS(serverHostnameOverride:)` when init swift-grpc client, looks like it still not work.
- About `openssl s_client -connect <host>:<port> -servername <host>` I use command `openssl s_client -connect hostname:443 -servername hostname` get below info:
<img width="1512" alt="截屏2021-12-07 01 40 10" src="https://user-images.githubusercontent.com/3955487/144894770-e8d8c128-c0a2-42fe-92ac-648fd9cc119a.png">

If I change command to `openssl s_client -connect hostname:443 -servername hostname:443` looks like every thing is fine:
<img width="889" alt="截屏2021-12-07 01 42 04" src="https://user-images.githubusercontent.com/3955487/144895023-33cb4bd9-f8b7-4a15-9c9f-8967612e1161.png">

[Lukasa]

This strongly suggests that your nginx config is wrong: it's expecting the wrong SNI header.

Can you add the server_name directive to your nginx server config? That is, change it to:

server {
                listen 443 ssl http2;
                server_name rpc.petpet.fun

                ssl_certificate ssl/domain.crt;
                ssl_certificate_key ssl/domain.key;

                location / {
                        # The 'grpc://' prefix is optional; unencrypted gRPC is the default
                        grpc_pass grpc://grpcservers;
                        # root   html;
                        # index  index.html index.htm;
                }
        }

Then try again?

---

About .withTLS(certificateVerification: .none) I think williamMillington you are right. It's not what I thought. It's just close the TLS channel.

Setting this does not disable TLS, it just turns off certificate verification. You're using it correctly.

[amazingyyc]

Hi Lukasa thanks for you quick reply, I modify my nignx server donfig to below:

Looks like I still cannot call success by hostname: my init code:

let channel = ClientConnection
                           // If I use usingPlatformAppropriateTLS with withTLS(certificateVerification: .none) will be crash.
                           //.usingPlatformAppropriateTLS(for: group)
                            .secure(group: group)
                            .withBackgroundActivityLogger(logger)
                            .withTLS(certificateVerification: .none)
                            // ip:address can success but hostname will fail.
                            .connect(host: "hostname", port: 443)
                            //.connect(host: "123.57.40.163", port: 443)

Blow is the error log by hostname:port.

2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 old_state=transientFailure new_state=connecting connectivity state change
2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 making client bootstrap with event loop group of type NIOTSEventLoop
2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 Network.framework is available and the EventLoopGroup is compatible with NIOTS, creating a NIOTSConnectionBootstrap
2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 connectivity_state=connecting activating connection
2021-12-07 01:55:20.540526+0800 PetPet[6443:984661] [tcp] tcp_input [C4.1:1] flags=[R] seq=1842261697, ack=0, win=0 state=ESTABLISHED rcv_nxt=1842261697, snd_una=1176676386
2021-12-07 01:55:20.541462+0800 PetPet[6443:984661] [connection] nw_read_request_report [C4] Receive failed with error "Connection reset by peer"
2021-12-07T01:55:20+0800 error gRPC : grpc.conn.addr_remote=123.57.40.163 grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 grpc.conn.addr_local=10.236.138.19 error=POSIXErrorCode: Connection reset by peer grpc client error
2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 connectivity_state=active deactivating connection
2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 delay_secs=4.21364450331115 scheduling connection attempt
2021-12-07T01:55:20+0800 debug gRPC : grpc_connection_id=A9A1BD03-025B-4CD4-90F1-17A15902FA0A/3 new_state=transientFailure old_state=connecting connectivity state change
2021-12-07 01:55:22.406297+0800 PetPet[6443:984665] [connection] nw_resolver_start_query_timer_block_invoke [C4] Query fired: did not receive all answers in time for rpc.petpet.fun:443

[amazingyyc]

Add .withTLS(serverHostnameOverride: "hostname:443") when init will request success by hostname. I donot know whether it is expected🤣🤣🤣. The code should be

let channel = ClientConnection
                            // .usingPlatformAppropriateTLS(for: group)
                            .secure(group: group)
                            .withBackgroundActivityLogger(logger)
                            .withTLS(certificateVerification: .none)
                            .withTLS(serverHostnameOverride: "hostname:443")
                            .connect(host: "hostname", port: 443)

[amazingyyc]

Looks like it's still cannot request without .withTLS(certificateVerification: .none), I will try some other config when initialize later..

[Lukasa]

Can you provide a link to your certificate? It seems like it's configured with the wrong hostname.

[amazingyyc]

Hi Lukasa Below is my sesrve pem content:

-----BEGIN CERTIFICATE-----
MIICqjCCAZICCQDkHsZStUi4ezANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDDApw
ZXRwZXQuZnVuMB4XDTIxMTIwNjE2MDIxNloXDTMxMTIwNDE2MDIxNlowGTEXMBUG
A1UEAwwOcnBjLnBldHBldC5mdW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQCrmXZrw/Glw6sCnsbWz6oSDaQM73x8xyR5u4rhO6wf3DXaCxlaWBn3VAuJ
ty10i+KIzK48afeQaT7FazZSnRLpynHRlHLG7vZIRwBgBWS+gc6o/3vvYQ0U8Ov9
0d0QwXbfCtH13qXDFFCmK8vBp1oAEYYAtBec3Mq2NomDUh8koUajMC38Ewh7WcLm
w/lVNoq3liaqsXFal4wGF7u9Tuwy5+HZm14bCM12bvJhg6/fA13KLuhXziDIquTB
YUtsdgaDaY/FO2UHrVR2CYble16YqY1yN0eEtNVkwK2ZCcKRUoF0RHWSg97vG9Fq
ZeNXIPWD1iMjKPwsiDXGFb+HfYEZAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAIdO
KhjpkAGJ9Skmkg0S72COODKIFvyrtNY/z3JkNR5Xgs9rvEncdCUwIQ4blhfwfJsQ
JpafrHOPklEwySxSqluc/GAVYeFOUua1zIPdJ6uoh7kEyhzDXsfGL2RuaicYbdk8
LikcK/wOb1JyTr8SmMMxsTgffUGvSFXZEGCW90gxAvv7vf9cWQJH8HvNqb/AIZ+w
PouCol40S4H01evrwVKLgYNmkqvQRVDmnqrlDKlK+UwSXWmhtxNjIthpuKpAOFZI
xVGIr/yp2vo033PpJ/IVZp2FfzEbY0HlowYJzLdrSMnosjb2XDa0j0mR7DWw6sBN
qXqpjYey5yXKDF7JxWg=
-----END CERTIFICATE-----

BTW I use below code it's also request success: I wondering Does it safe with withTLS(certificateVerification: .noHostnameVerification)?

let channel = ClientConnection
                            .secure(group: group)
                            .withBackgroundActivityLogger(logger)
                            .withTLS(certificateVerification: .noHostnameVerification)
                            .withTLS(serverHostnameOverride: "hostname:443")
                            .withTLS(trustRoots: NIOSSLTrustRoots.file(crtPath))
                            .connect(host: "hostname", port: 443)

[Lukasa]

Hmm, you don't have any subject alternative names in this certificate, which is a bit unfortunate. I wonder if updating the certificate to contain them would help. Otherwise I'm very confused: are we sure nginx is handling the TLS termination?

[amazingyyc]

Hmm, you don't have any subject alternative names in this certificate, which is a bit unfortunate. I wonder if updating the certificate to contain them would help. Otherwise I'm very confused: are we sure nginx is handling the TLS termination?

Thanks Lukasa I will try add SAN. BTW what do you mean: "nginx is handling the TLS termination"

[Lukasa]

BTW what do you mean: "nginx is handling the TLS termination"

Are we sure that nginx is the server process to which you're connecting?

[amazingyyc]

Yes I can find the connection log.