HTTP/2 support

[matzew]

Apple will have HTTP/2 API to send notification requests to APNs.

YAY! :tada:

Here is a little summary of more details:

• request/reponse (aka stream) for every sent (more reliable (e.g. JSON reason for a ‘bad request’ or 410 if the token is invalid))
  • allows ‘instant’ feedback (no separate feedback service!), since details are on the HTTP/2 response
• multiplexing: multiple requests (to APNs) on a single connection
• binary
• simpler certificate handling: Just a single cert! (no separate for dev/prod, VOIP etc)
• 4KB size of payload (for all versions of iOS/OSX), but just on the new HTTP/2 API

The new HTTP/2 API for APNs will be available in “Summer 2015″ for the development environment and will be made available for production “later this year”. No exact dates were given.

Details: https://developer.apple.com/videos/wwdc/2015/?id=720

[sunng87]

fuck yea!!!

[jchambers]

What an unreasonably huge relief ;)

Pushy is going to be a much, much simpler piece of software once this happens.

[jchambers]

I've asked how we can learn when the HTTP/2 docs become available, and I've been told that the best way to find out is to just keep monitoring developer.apple.com (I've asked for clarification as to whether the docs will appear in the main "Local and Remote Notification Programming Guide," or if they will appear in a separate document). I've been checking periodically, but please do post here if you see them before I do.

Thanks!

[matzew]

sounds like a plan

On Tue, Jun 30, 2015 at 5:40 AM, Jon Chambers notifications@github.com wrote:

I've asked how we can learn when the HTTP/2 docs become available, and I've been told that the best way to find out is to just keep monitoring developer.apple.com (I've asked for clarification as to whether the docs will appear in the main "Local and Remote Notification Programming Guide," or if they will appear in a separate document). I've been checking periodically, but please do post here if you see them before I do.

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/relayrides/pushy/issues/184#issuecomment-116934872.

Matthias Wessendorf

blog: http://matthiaswessendorf.wordpress.com/ sessions: http://www.slideshare.net/mwessendorf twitter: http://twitter.com/mwessendorf

[matzew]

@jchambers I think, w/ HTTP/2 it's no longer really needed to stick w/ Netty.

IMO something like OkHttpClient (supports HTTP/2) might be a big nicer to use

[jchambers]

w/ HTTP/2 it's no longer really needed to stick w/ Netty.

Yep. That's certainly a possibility, but I'll withhold judgment until we've actually seen the new docs.

[flozano]

this is huge... really looking forward to it.

what's wrong with netty? :)

[sunng87]

Netty has no plan to back port http2 stuff into current stable (4.0) branch. Perhaps we will wait for 4.1 to be stabilized?

[jchambers]

Some thinking out loud in the absence of additional details: I think affirmative acknowledgment of successful push notifications pretty much eliminates the need for most of the wacky internal queue stuff we were doing before. Without affirmative acknowledgment, it would have been impractical to use (for example) Futures to let callers know what happened to their notifications. Now we can probably just return a Future whenever somebody tries to send a notification and make life easier for everybody.

Really looking forward to seeing the details of this whole HTTP/2 thing when they come out.

[jchambers]

I emailed an evangelist at Apple. He said that they're still working on the HTTP/2 thing, but have no ETA.

[omarkilani]

All I can say is HOORAY.

The APNS protocol is a disaster (as shown by the millions of buggy implementations -- no fault of the developers of said implementations mind you). This has been a long time coming.

[matzew]

I emailed an evangelist at Apple. He said that they're still working on the HTTP/2 thing, but have no ETA.

too bad :-)

[jchambers]

I've opened #213 to track work in progress on HTTP/2 support.

[jchambers]

Hey, everybody! Looks like the updated APNs protocol has been published! See https://developer.apple.com/news/?id=12172015b for the announcement and https://developer.apple.com/library/ios/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/Chapters/Introduction.html for the details.

[omarkilani]

Let me be the first to say:

YEAAHH!!!!!!!!!!!

crosses fingers. Let's hope it actually works. :)

[sunng87]

:pray: for those countless hours we wasted in seeking for an unreceived push notification.

[jchambers]

It looks pretty straightforward, except for this bit about the headers:

To help avoid filling up this table and necessitating the discarding of table data, encode headers in the following way—especially when sending a large number of streams:

• The :path header should be encoded as a literal header field without indexing,
• The apns-id and apns-expiration headers should be encoded differently depending on initial or subsequent POST operation, as follows:
  • The first time you send these headers, encode them with incremental indexing to allow the header names to be added to the dynamic table
  • Subsequent times you send these headers, encode them as literal header fields without indexing

I haven't (yet) spotted a way to manage headers at quite that level in any of the clients I've looked at, but I also hadn't been looking very hard until now. I'm new to HTTP/2, tough; maybe that's just A Thing That Happens Automatically™? Suggestions are very welcome here!

[jchambers]

…also, in case anybody missed it, progress on the whole HTTP/2 thing lives at #213.

[jchambers]

…and I'm also scribbling down notes on the wiki as I go.

[omarkilani]

I don't think you actually need to do anything with headers. Pretty sure that's how HPACK works...

[jchambers]

Pretty sure that's how HPACK works...

That's mostly the conclusion I was coming to, too. It's a lower-priority thing, but I'll see if I can put together a coherent argument as to why that's true and send it along to the APNs team; they've been pretty receptive to feedback so far!

[omarkilani]

Well... I mean, it sounds like a "suggestion" anyway, but really, if they want to go HTTP/2, why saddle developers with more weird proprietary stuff -- isn't that the whole point of the new interface?

[omarkilani]

FWIW, I'm building a golang (1.6) implementation of the HTTP/2 APNS API. You get a lot of things for free there though (connection pooling, GOAWAY management, etc) so it's reasonably fast going.

I'll post if I run into any unexpected stuff. :)

[jchambers]

Hey, @omarkilani, I'm running into some certificate woes and am wondering how (if?) you're dealing with them. The docs say:

You can use your APNs certificate to send notifications to your primary app, as identified by its bundle ID, as well as to any Apple Watch complications or backgrounded VoIP services associated with that app. Use the (1.2.840.113635.100.6.3.6) extension in the certificate to identify the topics for your push notifications. For example, if you provide an app with the bundle ID com.yourcompany.yourexampleapp, you can specify the following topics in the certificate:

Extension ( 1.2.840.113635.100.6.3.6 )
Critical NO
Data com.yourcompany.yourexampleapp
Data app
Data com.yourcompany.yourexampleapp.voip
Data voip
Data com.yourcompany.yourexampleapp.complication
Data complication

I'd like to write some tests to make sure we're dealing with multi-topic certificates correctly, but am struggling to figure out how to actually generate a test certificate like that. To be clear, I can request a certificate with that extension from Apple, and the extension looks something like this:

ObjectId: 1.2.840.113635.100.6.3.6 Criticality=false
0000: 30 81 8E 0C 1D 63 6F 6D   2E 72 65 6C 61 79 72 69  0....com.relayri
0010: 64 65 73 2E 69 6F 73 2E   52 65 6C 61 79 52 69 64  des.ios.RelayRid
0020: 65 73 30 05 0C 03 61 70   70 0C 22 63 6F 6D 2E 72  es0...app."com.r
0030: 65 6C 61 79 72 69 64 65   73 2E 69 6F 73 2E 52 65  elayrides.ios.Re
0040: 6C 61 79 52 69 64 65 73   2E 76 6F 69 70 30 06 0C  layRides.voip0..
0050: 04 76 6F 69 70 0C 2A 63   6F 6D 2E 72 65 6C 61 79  .voip.*com.relay
0060: 72 69 64 65 73 2E 69 6F   73 2E 52 65 6C 61 79 52  rides.ios.RelayR
0070: 69 64 65 73 2E 63 6F 6D   70 6C 69 63 61 74 69 6F  ides.complicatio
0080: 6E 30 0E 0C 0C 63 6F 6D   70 6C 69 63 61 74 69 6F  n0...complicatio
0090: 6E                                                 n

…and Keychain Access certainly seems to know how to read that as being a string of Data entries:

…but using our actual production certificate for tests in an open-source library is, obviously, not going to fly.

keytool will happily add the extension by OID (via the -ext option—sorry there aren't anchors for a more specific link), but wants to us to provide a the "payload" for the extension as a hex dump. I'm not sure how to format things to get those Data entries to happen. I can back out a lot of the format (0x0c [length] [string]), but some pieces are still mysterious (0x30 ? ?). Is this a well-known format and I'm just googling for all the wrong things, or is there maybe something proprietary happening here?

tl;dr: How is Data formed? How certificate get extension?

[jchambers]

…a well-known format and I'm just googling for all the wrong things…

Yep. That's exactly what's going on. We're looking at DER encoding of ASN.1 types.

[omarkilani]

Hey @jchambers,

The strange thing about those Apple docs is that you can't really "request" specific data in the extension. It's always those three keys, which really doesn't make any sense... right?, because the APNS docs seem to say apns-topic is "optional".

But leaving it out is impossible due to how the certificate gets generated. So really you have to enforce always including apns-topic.

Re testing with multiple topic certs... Not really sure. How do you test a self-generated cert against the Apple endpoints anyway?

[omarkilani]

The reason I mention the above is that SimpleApnsPushNotification in http2 branch doesn't enforce specifying a topic, so I don't know how it's going to work?

[jchambers]

It's always those three keys, which really doesn't make any sense... right?, because the APNS docs seem to say apns-topic is "optional".

Well, old, pre-existing certificates won't have the topic extension, so we can get away without specifying an apns-topic in those cases.

So really you have to enforce always including apns-topic.

Hm. Should we do that before we're sure that all old-style certificates have expired and we're sure that new, single-topic certificates can't be generated? I could be convinced. It seems like, either way, we're creating a needless point of potential-wrongness for Pushy users.

If we require a topic for single-topic certificates, we're creating a situation where users aren't gaining anything, but have the opportunity to get the topic wrong. If we don't require a topic, we're creating an opportunity for users to fail to specify one when they actually need it. Hmph.

How do you test a self-generated cert against the Apple endpoints anyway?

Can't ;)

What we've been doing is using a mock APNs server for testing, and using custom a custom TrustManager that expects our test certificates, but no others.

Of course, we do limited testing with our production certificate and actual phones, too, but that's more of a thing I grumble about and do manually before releases.

[omarkilani]

Does the http2 branch currently differentiate between "old style" and "new style" certs?

It seems to me that the new branch without a PushManager etc would be the perfect time to enforce specifying a topic with each payload, no? Since the topic is required for "new style" certs, but optional for "old style" certs, requiring it in the pushy API seems like the only possible solution? :)

[jchambers]

Does the http2 branch currently differentiate between "old style" and "new style" certs?

No. In the past, that wasn't possible because given a KeyStore, we wouldn't know what credentials the client would actually present to the server. If we're going to attempt native OpenSSL support, though, we'll need to structure things differently (i.e. take an explicit X509Certificate instead of a full KeyStore), and so it may make sense to start sanity-checking certificates at launch time. Still, it seems like one of those cases where we could build a lot of really complex machinery to say, "hey, this might not work," when the user would just find out the same thing by actually attempting a connection. I'll think about it, though.

Since the topic is required for "new style" certs, but optional for "old style" certs, requiring it in the pushy API seems like the only possible solution?

I'm not sure if we're on the same page. It's true that we don't require it right now, but we do allow it. Is that what you mean?

In any case, I think I'm coming around to the idea that we shouldn't have constructors that don't take a topic as an argument. If callers really want to, they can pass in null for a topic and we'll handle that gracefully.

[omarkilani]

Sorry if it wasn't clear, but as far as I can see, attempting to send a payload with a "new style" cert without specifying a apns-topic results in BadTopic because certs are (now) always generated with 3 topics.

At least that's what happens to me...?

So let's pretend you're a new pushy user or whatever and you're not following this issue. You attempt to send your first payload with your fancy new cert, and it fails instantly with BadTopic. You spend X hours investigating what that means... :)

[jchambers]

Yep. Makes sense.

Like I said: I think I'm sold on "promoting" the topic in the constructor list. Will take a stab at that shortly.

[jchambers]

@omarkilani topic is an argument for all SimpleApnsPushNotification constructors as of b5c4b72. Docs still need some help, but I'm planning on making a holistic documentation pass soon.

[omarkilani]

I do find it kind of odd that the APNS endpoint doesn't just default to the 'primary' cert topic encoded in the extension if apns-topic isn't specified.

Especially since .voip and .complication are added even if you have no VoIP or WatchKit stuff enabled.

The only explanation that makes sense is that they want people to start sending apns-topic all the time, and this is the first step in deprecating the 'default topic' ability.

[jchambers]

@omarkilani Call it Stockholm syndrome, but this is a level of odd I'm happy to accept ;)

I should mention that the engineering crew at Apple has been pretty receptive to feedback about the HTTP/2 stuff so far. I'll certainly mention it to them, and I'm sure it wouldn't hurt if you did the same.

[omarkilani]

I don't actually have any problems with it -- I'm all for the explicit apns-topic.

I just think the docs are misleading in that they say it's "optional" when really that's only the case for "old style" certs. The first thing I did when reading the new APNS HTTP/2 docs was generate a "new style" cert since it simplifies production/sandbox stuff significantly.

The real issue (unrelated to pushy) is this: say I'm on an "old style" cert and my servers are happily pushing payloads with no apns-topic defined. It's about to expire and my ops team or whatever gets the ticket to renew the cert, so they go to developer.apple.com and do the cert renewal as usual. The new cert gets generated with the 3 embedded topics. All of a sudden nothing works and they get BadTopic for everything. Someone reads the docs, reads "optional" apns-topic and skips over the entire thing.

Anyway, that's just ranting at this point. I'm glad pushy is now enforcing it. :)

I'm going to deploy the http2 branch into fairly heavy production use in the next day or so on a not-so-critical component and will report how things go. :)

[jchambers]

I'm going to deploy the http2 branch into fairly heavy production use in the next day or so on a not-so-critical component and will report how things go. :)

Awesome! Please do let us know!

One thing I should be clear about: for now, we ONLY do Java's SSL provider. That means you'll need to have alpn-boot on your boot classpath as described here: http://www.eclipse.org/jetty/documentation/current/alpn-chapter.html

alpn-boot will only work for OpenJDK 7 and 8. You'll also need some ciphers that aren't available before Java 8, so Java 8 is pretty much a requirement right now.

By the time we ship, we'll support native OpenSSL in addition to Java's own SSL provider. When that's in place, everything should work all the way back to Java 6, and hopefully with less hoop-jumping. That's not the case yet, though.

[omarkilani]

Hey @jchambers,

Any plans to add more logging/tracing into the http2 branch?

I'm trying to figure out why it's locking up under load... and why some messages are being lost.

It's hard to do it without running in production though -- works alright at a small requests/second rate.

[jchambers]

Any plans to add more logging/tracing into the http2 branch?

Yes, but probably not for a couple days.

One thing you might do for now is to revert a5f240692868b87f9d3a7baf271b669e7e8b8aee, which removed HTTP/2 frame loggers.

[jchambers]

@omarkilani The lock-up seems to be happening in cases where we hit the limit for the maximum number of open streams. I haven't identified the root cause yet, but can reproduce the issue reliably with our mock server.

[jchambers]

can reproduce the issue reliably with our mock server.

I lied. I was able to reproduce an issue with our mock server with our mock server ;)

I'm not actually sure what might be wrong in production, but hope more logging will help identify the cause.

[jchambers]

@omarkilani Restored logging in 67c907c, if you want to give it another go.

[omarkilani]

@jchambers I'm currently unable to build the http2 branch. Same build failure as https://travis-ci.org/relayrides/pushy

[jchambers]

Oops. Fixed. Sorry about that!

[omarkilani]

@jchambers Thanks for the quick fix. Am I missing something obvious regarding multiple connections to the APNS servers or is that now something for the library end user to look after?

[jchambers]

Am I missing something obvious regarding multiple connections to the APNS servers or is that now something for the library end user to look after?

@omarkilani this will be an overkill answer for you if you're writing your own implementation, but I'll go into detail for the more casual parts of the audience ;)

Ultimately, it's something I'd like to restore, but I don't think it will happen for the initial HTTP/2 release.

In the past, connections would close whenever the gateway rejected a notification, which was quite often. In some cases, connections would spend more connecting/handshaking than they would actually sending notifications. Under those circumstances, it made sense to have a "spinning reserve" of connections so sending could continue even when one connection dropped.

With HTTP/2, connections are much less likely to close since connections stay open even when notifications are rejected. There are still reasons to keep multiple connections open, though. According to the docs:

You may establish multiple connections to APNs servers. If you need to send a large number of remote notifications, spread them out over connections to several different gateways. This improves performance compared to using a single connection, in that it lets you send the remote notifications faster, and it lets APNs deliver them faster.

The need for redundancy is gone now, but there's an opportunity to do some load balancing. To take advantage of that I think we need to do a few things:

1. Do some fancier-than-usual DNS stuff to learn how many servers are in play and what their IPs are. This isn't a hard thing for DNS to do in general, but it will require—I think—some extra hoop-jumping to make it happen in Java.
2. Figure out a connection pooling/rotation mechanism. I think Netty's ChannelPool is what we need, but there are some details I haven't figured out yet.
3. Do fancier things to manage multiple connections at client startup/shutdown.
4. Make sure that the performance cost of rotating through connections doesn't outweigh the gains from sending on multiple channels.

So, in short, I think it's doable, but will be a reasonably sizable project. Most users should fare as well with a single connection with HTTP/2 as they were with multiple connections with previous versions of the APNs protocol, and we'll hope to restore (add, really—previous implementations only did load balancing by accident) parallel connections for our more industrial users.

[omarkilani]

Alright, makes sense. I do see the connection terminated pretty regularly on the HTTP/2 API, so that was kind of the reason behind the question. :)

I wonder if this

        return SslContextBuilder.forClient()
                .sslProvider(OpenSsl.isAlpnSupported() ? SslProvider.OPENSSL : SslProvider.JDK)
                .ciphers(Http2SecurityUtil.CIPHERS, SupportedCipherSuiteFilter.INSTANCE)
                .keyManager(certificate, privateKey)
                .trustManager(CA_CERTIFICATE)
                .applicationProtocolConfig(new ApplicationProtocolConfig(Protocol.ALPN,
                        SelectorFailureBehavior.NO_ADVERTISE,
                        SelectedListenerFailureBehavior.ACCEPT,
                        ApplicationProtocolNames.HTTP_2))
                .build();

Should be part of the codebase somehow. I know SslContextUtil went away... but if that code is needed to establish a connection, it seems like it should be part of pushy somehow.

[omarkilani]

Ah, whoops. ApnsClient constructors changed. Nevermind.

Hmmm.... in that case, it would be nice to be able to supply a p12 and a password. Is that incompatible with the topic parsing stuff?

[jchambers]

I do see the connection terminated pretty regularly on the HTTP/2 API

Interesting. That reminds me that we should add logging when we get GOAWAY frames so we know WHY the connection is closing.

it would be nice to be able to supply a p12 and a password.

Strongly agreed, but I haven't yet figured out how to extract the private key from a p12. In fairness, I kinda waved my hands at it with keytool and didn't dig much deeper, so it's very likely there's a good way to do it I just haven't figured out yet. I'll give it another look (maybe with an actual KeyStore?) shortly; I was planning on revisiting the ApnsClient constructors anyhow to see if we can figure out some kind of "managed mode" where callers don't need to provide an EventLoopGroup, so I'll probably try to add a p12 constructor while I'm in there.

[omarkilani]

Alright -- it's running again in production.

So the issue seems to be that Apple just drops the connection for no reason:

TRACE [2016-01-06 21:37:24,650] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClientHandler - Received headers from APNs gateway on stream 1357: DefaultHttp2Headers[:status: 200, apns-id: 27CFDC15-E305-27E0-E290-8FC98E115899]
DEBUG [2016-01-06 21:37:24,650] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClient - Received response from APNs gateway: SimplePushNotificationResponse [pushNotification=SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116243519"}}, invalidationTime=Thu Jan 07 21:37:24 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm], success=true, rejectionReason=null, tokenExpirationTimestamp=null]
TRACE [2016-01-06 21:37:25,173] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClientHandler - Wrote headers on stream 1365: DefaultHttp2Headers[:method: POST, :path: /3/device/X, content-length: 60, apns-expiration: 1452202645, apns-priority: 10, apns-topic: com.rememberthemilk.mobilertm]
TRACE [2016-01-06 21:37:25,173] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClientHandler - Wrote payload on stream 1365: {"aps":{"content-available":1},"rtm":{"ts":"1452116244154"}}
TRACE [2016-01-06 21:37:25,173] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClientHandler - Wrote headers on stream 1367: DefaultHttp2Headers[:method: POST, :path: /3/device/X, content-length: 60, apns-expiration: 1452202645, apns-priority: 10, apns-topic: com.rememberthemilk.mobilertm]
TRACE [2016-01-06 21:37:25,174] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClientHandler - Wrote payload on stream 1367: {"aps":{"content-available":1},"rtm":{"ts":"1452116244154"}}
TRACE [2016-01-06 21:37:25,265] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClientHandler - Received headers from APNs gateway on stream 1365: DefaultHttp2Headers[:status: 200, apns-id: 3437F18F-7699-3156-117B-F1390EB313B9]
DEBUG [2016-01-06 21:37:25,265] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClient - Received response from APNs gateway: SimplePushNotificationResponse [pushNotification=SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116244154"}}, invalidationTime=Thu Jan 07 21:37:25 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm], success=true, rejectionReason=null, tokenExpirationTimestamp=null]
DEBUG [2016-01-06 21:37:25,266] [nioEventLoopGroup-2-2] io.netty.handler.codec.http2.Http2ConnectionHandler - Sent GOAWAY: lastStreamId '0', errorCode '2', debugData ''. Forcing shutdown of the connection.
DEBUG [2016-01-06 21:37:25,267] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClient - Disconnected. Next automatic reconnection attempt in 1 seconds.
DEBUG [2016-01-06 21:37:25,503] [default-akka.actor.default-dispatcher-5] com.relayrides.pushy.apns.ApnsClient - Failed to send push notification because client is not connected: SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116244000"}}, invalidationTime=Thu Jan 07 21:37:25 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm]
DEBUG [2016-01-06 21:37:25,523] [default-akka.actor.default-dispatcher-5] com.relayrides.pushy.apns.ApnsClient - Failed to send push notification because client is not connected: SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116244000"}}, invalidationTime=Thu Jan 07 21:37:25 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm]
DEBUG [2016-01-06 21:37:26,268] [nioEventLoopGroup-2-2] com.relayrides.pushy.apns.ApnsClient - Attempting to reconnect.
DEBUG [2016-01-06 21:37:26,473] [default-akka.actor.default-dispatcher-13] com.relayrides.pushy.apns.ApnsClient - Failed to send push notification because client is not connected: SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116245458"}}, invalidationTime=Thu Jan 07 21:37:26 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm]
DEBUG [2016-01-06 21:37:26,473] [default-akka.actor.default-dispatcher-2] com.relayrides.pushy.apns.ApnsClient - Failed to send push notification because client is not connected: SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116245458"}}, invalidationTime=Thu Jan 07 21:37:26 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm]
DEBUG [2016-01-06 21:37:26,473] [default-akka.actor.default-dispatcher-12] com.relayrides.pushy.apns.ApnsClient - Failed to send push notification because client is not connected: SimpleApnsPushNotification [token=X, payload={"aps":{"content-available":1},"rtm":{"ts":"1452116245458"}}, invalidationTime=Thu Jan 07 21:37:26 UTC 2016, priority=IMMEDIATE, topic=com.rememberthemilk.mobilertm]
DEBUG [2016-01-06 21:37:26,652] [nioEventLoopGroup-2-3] io.netty.handler.ssl.SslHandler - [id: 0x0cec87c7, /108.168.199.157:39650 => api.push.apple.com/17.143.164.138:443] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
INFO [2016-01-06 21:37:26,654] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClient - Connection to api.push.apple.com/17.143.164.138:443 restored.
TRACE [2016-01-06 21:37:27,363] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClientHandler - Wrote headers on stream 1: DefaultHttp2Headers[:method: POST, :path: /3/device/X, content-length: 60, apns-expiration: 1452202647, apns-priority: 10, apns-topic: com.rememberthemilk.mobilertm]
TRACE [2016-01-06 21:37:27,363] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClientHandler - Wrote payload on stream 1: {"aps":{"content-available":1},"rtm":{"ts":"1452116246000"}}
TRACE [2016-01-06 21:37:27,363] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClientHandler - Wrote headers on stream 3: DefaultHttp2Headers[:method: POST, :path: /3/device/X, content-length: 60, apns-expiration: 1452202647, apns-priority: 10, apns-topic: com.rememberthemilk.mobilertm]
TRACE [2016-01-06 21:37:27,363] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClientHandler - Wrote payload on stream 3: {"aps":{"content-available":1},"rtm":{"ts":"1452116246000"}}
TRACE [2016-01-06 21:37:27,363] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClientHandler - Wrote headers on stream 5: DefaultHttp2Headers[:method: POST, :path: /3/device/X, content-length: 60, apns-expiration: 1452202647, apns-priority: 10, apns-topic: com.rememberthemilk.mobilertm]
TRACE [2016-01-06 21:37:27,363] [nioEventLoopGroup-2-3] com.relayrides.pushy.apns.ApnsClientHandler - Wrote payload on stream 5: {"aps":{"content-available":1},"rtm":{"ts":"1452116246000"}}

And the messages that are attempted when the connection isn't connected are just dropped.

... So it looks like we're back to connection pools, queues, and what not.... :)