Closed GoogleCodeExporter closed 9 years ago
Could you please be more specific about what you mean by "lost messages"? Do
you mean that the messages added to the queue are never sent by the library, or
that the library does send the messages but they never reach the devices?
Original comment by sype...@gmail.com
on 13 Apr 2012 at 5:25
the messages added to the queue but never reach the devices.
I fixed this problem by close the connection(to apn server) after idle for more
then 5 minutes. I found when the connection idle for a long time. the new
message maybe can't reach the devices.
Original comment by minkon1...@gmail.com
on 17 Apr 2012 at 6:42
I am unable to reproduce the issue. Could you please attach a stack trace dump
for when the queue has messages to deliver but apparently is not sending them?
This will help understand what the library is doing... Thank you!
Original comment by sype...@gmail.com
on 21 Apr 2012 at 4:47
I have the same problem (I am using a Queue. Sometimes my messages are lost
with no explanation, and everything seems to be working fine.
Original comment by oli...@crowdplayce.com
on 26 Apr 2012 at 7:50
Unfortunately, without more details, I cannot reproduce the issue. Could you
please attach a stack trace dump for when the queue has messages to deliver but
apparently is not sending them?
Original comment by sype...@gmail.com
on 26 Apr 2012 at 9:47
Issue 118 has been merged into this issue.
Original comment by sype...@gmail.com
on 26 Apr 2012 at 9:49
Original comment by sype...@gmail.com
on 2 May 2012 at 1:44
I have the exact same problem. The problem is NOT that the message gets stuck
in the queue. The queue always send the message according to the log. That is,
I get
DEBUG [javapns.notification.PushNotificationManager] - At this point, the
entire 191-bytes message has been streamed out successfully through the SSL
connection
but the message never reaches the device. No errors ever show up in the log.
This seems to happen after the connection has been idle for some time.
A restart of the queue fixes the problem.
If I do
PushedNotifications notifications = queue.getPushedNotifications();
Iterator<PushedNotification> iter = notifications.iterator();
while(iter.hasNext())
{
PushedNotification pn = iter.next();
log.debug("success: " + pn.isSuccessful());
log.debug("PN exception: ", pn.getException());
}
every PushedNotification gets success: true and no exceptions.
Original comment by fredrik....@gmail.com
on 3 May 2012 at 8:06
#8 thats exactly what happens to me. The connection becomes idle and it stops
working
Original comment by oli...@crowdplayce.com
on 3 May 2012 at 8:42
Once I see this issue (where the message is reported delivered but never
actually makes it) I get the following for successive notifications:
--
2012-05-07 08:14:04 PushNotificationManager [DEBUG] Building Raw message from
deviceToken and payload
2012-05-07 08:14:04 PushNotificationManager [DEBUG] Built raw message ID
16777262 of total length 171
2012-05-07 08:14:04 PushNotificationManager [DEBUG] Attempting to send
notification: <PAYLOAD>
2012-05-07 08:14:04 PushNotificationManager [DEBUG] to device: <TOKENID>
--
..and that's the last I hear from that push. No failures, nothing.
Original comment by jcost...@gmail.com
on 7 May 2012 at 8:55
As I asked a few times in this thread, a stack trace dump is needed to
investigate the issue. Otherwise nothing is being done on this issue.
Original comment by sype...@gmail.com
on 7 May 2012 at 1:29
hi sype!
i'm not sure i understand what you want me to provide. could you be more
specific?
the messages do get sent so the queue is empty and there are no error stack
traces.
in fact, i get a positive response saying that notification was sent on first
attempt.
Original comment by fredrik....@gmail.com
on 15 May 2012 at 8:15
a stack trace dump, which is a basic debugging operation for capturing the
state of all threads in a Java program...
(http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/)
Original comment by sype...@gmail.com
on 15 May 2012 at 8:41
i have the exact same problem on the same server (amazon linux). i am also
looking for a solution.
@sype: i would have provided the dump stacktrace you ask if you could tell us
at what moment you want it (before adding into queue, during the push -i don't
know what to look for-, after the push) ??
Original comment by hguve...@gmail.com
on 25 May 2012 at 12:24
I have found something about the dropping connections regarding servers on
Amazon;
https://forums.aws.amazon.com/thread.jspa?threadID=33427&start=0&tstart=0
Long story short: We cannot use PushQueue as it is implemented at the moment.
It needs to send heartbeat data if it is inactive for less then every 60
seconds.
Original comment by hguve...@gmail.com
on 25 May 2012 at 12:46
i have looked into this connection drop issue and want to confirm it on my
server Amazon Linux on micro instance EC2 ... this is the result
[ec2-user@XXX ~]$ lsof | grep TCP
java 14893 ec2-user 375u IPv6 375102 0t0 TCP
*:8181 (LISTEN)
java 14893 ec2-user 376u IPv6 375103 0t0 TCP
*:webcache (LISTEN)
java 14893 ec2-user 381u IPv6 375107 0t0 TCP
*:appserv-http (LISTEN)
java 14893 ec2-user 385u IPv6 375109 0t0 TCP
*:lrs-paging (LISTEN)
java 14893 ec2-user 389u IPv6 375111 0t0 TCP
*:imqbrokerd (LISTEN)
java 14893 ec2-user 462u IPv6 478553 0t0 TCP
XXX.compute-1.internal:50020->st11p01st-interface003-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 465u IPv6 478554 0t0 TCP
XXX.compute-1.internal:50021->st11p01st-interface003-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 468u IPv6 376504 0t0 TCP
*:sun-as-jmxrmi (LISTEN)
java 14893 ec2-user 500u IPv6 442333 0t0 TCP
XXX.compute-1.internal:52537->st11p01st-interface010-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 503u IPv6 442347 0t0 TCP
XXX.compute-1.internal:36621->st11p01st-interface009-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 519u IPv6 455136 0t0 TCP
XXX.compute-1.internal:40070->st11p01st-interface009-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 520u IPv6 477479 0t0 TCP
XXX.compute-1.internal:42686->st11p01st-interface001-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 521u IPv6 477480 0t0 TCP
XXX.compute-1.internal:42687->st11p01st-interface001-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 522u IPv6 477495 0t0 TCP
XXX.compute-1.internal:51542->nk11p01st-interface003-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 527u IPv6 455120 0t0 TCP
XXX.compute-1.internal:34270->st11p01st-interface015-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 530u IPv6 455121 0t0 TCP
XXX.compute-1.internal:34271->st11p01st-interface015-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 534u IPv6 455135 0t0 TCP
XXX.compute.compute-1.internal:40069->st11p01st-interface009-bz.push.apple.com:2
195 (CLOSE_WAIT)
java 14893 ec2-user 535u IPv6 478594 0t0 TCP
XXX.compute-1.internal:34723->st11p01st-interface015-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 543u IPv6 478673 0t0 TCP
XXX.compute-1.internal:57461->nk11p01st-interface014-bz.push.apple.com:2195
(CLOSE_WAIT)
java 14893 ec2-user 544u IPv6 478686 0t0 TCP
XXX.compute-1.internal:33197->st11p01st-interface012-bz.push.apple.com:2195
(CLOSE_WAIT)
Original comment by hguve...@gmail.com
on 25 May 2012 at 1:14
this is really irritating to say that but the Queue implementation doesn't
provide production requirements for any project...
this is the bottom line issue of the implementation of flushing data into
CLOSE_WAIT connection;
http://code.google.com/p/javapns/source/browse/tags/2.2/src/javapns/notification
/PushNotificationManager.java#416
and this is the issue has been talked on stackoverflow
http://stackoverflow.com/questions/2028620/java-sockets-and-dropped-connections
the connection has to be closed and open on every push notification is being
released.
I hoped this can help anyone who suffered with this...
Original comment by hguve...@gmail.com
on 25 May 2012 at 1:38
[deleted comment]
the stackoverflow link is about an issue for apns on github... different
projects same issue...
Original comment by hguve...@gmail.com
on 25 May 2012 at 1:41
My server just stopped sending.. Here is stack trace (parts of it).. No
deadlocks
Thread 1207: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- javapns.notification.transmission.NotificationThread.runQueue() @bci=192, line=279 (Compiled frame)
- javapns.notification.transmission.NotificationThread.run() @bci=44, line=202 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 1206: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- javapns.notification.transmission.NotificationThread.runQueue() @bci=192, line=279 (Compiled frame)
- javapns.notification.transmission.NotificationThread.run() @bci=44, line=202 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 28448: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run() @bci=13, line=141 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 28447: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame)
- org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(java.net.ServerSocket) @bci=1, line=59 (Compiled frame)
- org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run() @bci=81, line=210 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 28446: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run() @bci=22, line=1378 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 28441: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- sun.misc.GC$Daemon.run() @bci=51, line=100 (Interpreted frame)
Thread 28436: (state = BLOCKED)
Thread 28435: (state = BLOCKED)
Thread 28434: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame)
Thread 28433: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
Thread 28426: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=408 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=462 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=430 (Interpreted frame)
- org.apache.catalina.core.StandardServer.await() @bci=175, line=447 (Interpreted frame)
- org.apache.catalina.startup.Catalina.await() @bci=4, line=709 (Interpreted frame)
- org.apache.catalina.startup.Catalina.start() @bci=186, line=654 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame)
- org.apache.catalina.startup.Bootstrap.start() @bci=37, line=322 (Interpreted frame)
- org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=135, line=450 (Interpreted frame)
Original comment by oli...@crowdplayce.com
on 20 Jun 2012 at 2:37
My little 5cents:
I have the same problems on production servers. The problems typically arise
after a given (long) uptime of the server.
My analysis:
There is probably on the datacenter a network equipment that silently close the
socket between my server and the apple servers.
Because there is no ack on the push message sent to apple servers, it takes a
reda on the socket or a close on the socket for the javapns queue to discover
that the socket is actually closed. Alas, the messages is already sent and
confirmed sent (xxx bytes has been flushed blablabla).
This is confirmed because, while reading the logs while sending various
messages, the loss never happens when the server connection is detected as down
and renewed.
In order to solve this issue I have implemented a "reopen the socket for each
messages" policy (however this must slow down the queue to a crawl, I don't
know, I have no load).
AppleNotificationServer server = new AppleNotificationServerBasicImpl(config.getKeystore(), config.getPassword(), ConnectionToAppleServer.KEYSTORE_TYPE_JKS , isProduction());
List<NotificationThread> allThreads = new ArrayList<>();
for (int i=0;i<config.getNumberOfThreads();i++) {
PushNotificationManagerWithSocketReopenEachTime notifManager = new PushNotificationManagerWithSocketReopenEachTime(server);
NotificationThread thread = new NotificationThread(null,notifManager,server);
allThreads.add(thread);
}
queue = new NotificationThreads(server, null, allThreads);
config.isProduction(), config.getNumberOfThreads());
queue.start();
}
public static class PushNotificationManagerWithSocketReopenEachTime extends PushNotificationManager {
AppleNotificationServer server;
public PushNotificationManagerWithSocketReopenEachTime(AppleNotificationServer server) {
super();
this.server=server;
}
@Override
public PushedNotification sendNotification(Device device, Payload payload, boolean closeAfter, int identifier) throws CommunicationException {
try {
restartConnection(server);
} catch (KeystoreException e) {
throw new FizRuntimeException("cannot connect to apns server with keystore",e);
}
return super.sendNotification(device, payload, closeAfter, identifier);
}
}
Original comment by jerome.b...@gmail.com
on 20 Jun 2012 at 2:47
Hey Jerome
Cool solution.. How about some time policy. Because it seems that this only
happens when a connection is unused. So in order to preserve performance we
reopen unused connections?. Opening a connections takes several seconds. I
discovered that when I tried todo the simple one connection pr message
I have quite a bit of traffic and the connections stopped working one by one
until I got no push notifications sent at all. But I need the queue in order to
be able to scale my solution.
Btw Jerome... Whats this. I could not figure out where it belonged and there
was a } too much?
config.isProduction(), config.getNumberOfThreads());
public class PushNotificationManagerWithSocketReopenEachTime extends
javapns.notification.PushNotificationManager {
AppleNotificationServer server;
public static final long unusedConnectionToleranceMs = 5000;
private long lastUsage = 0;
public PushNotificationManagerWithSocketReopenEachTime(AppleNotificationServer server) {
super();
this.server=server;
}
@Override
public PushedNotification sendNotification(Device device, Payload payload, boolean closeAfter, int identifier) throws CommunicationException {
if(System.currentTimeMillis() - lastUsage > unusedConnectionToleranceMs) {
try {
restartConnection(server);
} catch (KeystoreException e) {
throw new RuntimeException("cannot connect to apns server with keystore",e);
}
}
lastUsage = System.currentTimeMillis();
return super.sendNotification(device, payload, closeAfter, identifier);
}
}
Original comment by oli...@crowdplayce.com
on 21 Jun 2012 at 6:29
Original comment by sype...@gmail.com
on 2 Jul 2012 at 10:33
I have the same problem and I'm not sure when it would be neccesary to reopen a
connection. To reopen connections it's a very expensive operation and depending
the kind of the message sent, it's possible the time spent were unacceptable.
I don't need to send too many messages per hour, but I do need they arrive very
fast to the device.
I'll try minkon1's solution.
Original comment by manusd...@gmail.com
on 31 Aug 2012 at 10:32
[deleted comment]
Any update on this ?
We are facing similar issue in production too. Do let us if solution provided
in threads, fixes the issue
Original comment by dksid...@gmail.com
on 1 Mar 2013 at 1:11
My fix worked, stable for 6 months
Original comment by olive...@gmail.com
on 1 Mar 2013 at 3:13
I have the same problem and I'm not sure when it would be success to reopen a
connection.Because I found that some time the new message still can't reach the
devices.
Original comment by szf881...@163.com
on 28 Mar 2013 at 9:14
2013-03-28 19:24:05,529 DEBUG javapns.communication.ConnectionToAppleServer -
Creating SSLSocketFactory
2013-03-28 19:24:05,589 DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to gateway.sandbox.push.apple.com:2195
2013-03-28 19:24:06,110 DEBUG javapns.notification.PushNotificationManager - Initialized Connection to Host: [gateway.sandbox.push.apple.com] Port: [2195]: 1a93f38[SSL_NULL_WITH_NULL_NULL: Socket[addr=gateway.sandbox.push.apple.com/17.149.34.65,port=2195,localport=40485]]
2013-03-28 19:24:06,160 DEBUG javapns.notification.PushNotificationManager - Building Raw message from deviceToken and payload
2013-03-28 19:24:06,170 DEBUG javapns.notification.PushNotificationManager - Built raw message ID 1 of total length 97
2013-03-28 19:24:06,170 DEBUG javapns.notification.PushNotificationManager - Attempting to send notification: {"aps":{"alert":"xinfeng000000","badge":1},"type":1}
2013-03-28 19:24:06,170 DEBUG javapns.notification.PushNotificationManager - to device: 42e6dce2612ab5f154e6feecc364a7b7415f7ff0e749aadd6e5cecd5f9b32774
2013-03-28 19:24:14,882 DEBUG javapns.notification.PushNotificationManager - Flushing
2013-03-28 19:24:14,882 DEBUG javapns.notification.PushNotificationManager - At this point, the entire 97/172.18.10.73 gateway.sandbox.push.apple.com/17.149.34.65-bytes message has been streamed out successfully through the SSL connection
2013-03-28 19:24:14,882 DEBUG javapns.notification.PushNotificationManager - Notification sent on first attempt
Original comment by szf881...@163.com
on 28 Mar 2013 at 11:27
Implemented feature in comment 22 in r391.
By default, inactivity of 60 seconds will cause a connection restart before the
next payload is sent.
Inactivity delay can be changed via
PushNotificationManager.setRestartConnectionAfterInactivity(ms)
Original comment by sype...@gmail.com
on 30 Sep 2014 at 4:00
Original issue reported on code.google.com by
minkon1...@gmail.com
on 12 Apr 2012 at 8:11