Ghost Notifcations

[trevorrjohn]

Hey, so I am seeing this weird issue where I will get a notification saying "The site was updated in the background." It seems to be shown when other push messages are coming in, however this notification has no icon and is not clickable.

Screenshot:

I am not sure if this is coming from the server or if it being displayed by the web-sdk. Any thoughts?

[froodian]

Hi @trevorrjohn,

Unfortunately this message is something that the browser creates and displays on its own in certain situations, notably when it doesn't think that we're displaying our own notification (which, obviously we are, given that the other notification is on-screen simultaneously).

I've generally only see this occur with registrations created on older browsers (which didn't support payloads natively, and had to resort to what I call "tickle-and-fetch", where the browser receives a dataless "tickle" from the server, and then needs to "fetch" the actual push notification). In that scenario, if two pushes are sent in rapid succession and then the browser tries to fetch both simultaneously, this can occur.

In this case, what version of the browser is installed? What version was installed when the push registration occurred? Is this happening with all pushes you receive, or only occasionally? When you're seeing this, were multiple pushes recently sent in rapid succession to that user? I ask these questions to try to determine if this is like the scenario I describe above, or if there's some other edge case that is causing the browser to determine that we're not displaying a foreground push that we received in certain scenarios.

If you're using Chrome, I'd also be interested in what you see if you navigate to chrome://serviceworker-internals/ - do you see only one entry for your site, and if so, what are the "Installation Status"/"Running Status"/"Script" values for that entry? Is there anything in the "Log" area?

If you're more comfortable answering these questions privately, feel free to follow up via desk ticket, just mention my name when you file it and it'll come back around to me.

[trevorrjohn]

Hi @froodian,

Thanks for the info.

This is occurring in the latest version of Chrome (53.0.2785.143). I believe that this is the version installed when push registration occurred. This is just happening occasionally. It seems that it happens when opening Chrome for the first time in the morning or when turning on my computer, although not all the time.

Installation Status: ACTIVATED
Running Status: RUNNING
Version ID: 1117
Renderer process ID: 29007
Script: https://<domain>.com/service-worker.js 
Renderer thread ID: 28171
DevTools agent route ID: 3

There was nothing in the log area, but if I see the notification again I will go back and check again.

Just to note, one of my teammates also saw this notification and he was using Vivaldi. (Not a big concern, but thought I should let you know.)

Let me know if there is any other info that would be helpful.

[trevorrjohn]

Just to follow up here, I just saw another ghost notification. I check the logs, but nothing was showing.

[froodian]

Thanks for the info @trevorrjohn,

I'm sorry to hear that the registration is so recent, this is the first confirmation I've had of this issue still occurring under current conditions.

The "first time in the morning" info is particularly interesting, to muse for a second, I wonder if this has to do somehow with a push being sent while the machine is offline, with no Chrome helper process running, or the network being down - or maybe with multiple push being sent under those conditions, and then both getting delivered simultaneously when the process wakes up or network connectivity is restored.

We'll spend some time digging into this and look for a more reliable way to consistently suppress these notifications.

[trevorrjohn]

Hey @froodian any update on this? We are seeing this quite a bit on all of our sites at the moment. Still not seeing any errors in the console.

[froodian]

Hi @trevorrjohn,

Unfortunately we have not had much luck reproducing this issue on our end. I've done some research and debugging, and it appears the general conditions that could cause this ghost push are:

• The "tickle-and-fetch" scenario I described earlier - doesn't seem to be the issue here since these are subscriptions created with recent Chrome. Regardless I'd like to add a little server-side logging on our end for one of your users specifically impacted to be entirely sure that we're correctly recognizing the subscription as supporting payload push.
• A third-party service worker (as opposed to Appboy's service worker) is also registered for your site. I don't expect that this is the scenario here, as you'd see this service worker on the chrome://serviceworker-internals/ page, but I wanted to call it out for completeness' sake - and if you see more than one entry on that page for your site, definitely mention that.
• A bug in Appboy's service worker is causing the promise not to be resolved in some circumstances. This seems like a likely culprit at this point, but we've had multiple people audit this file looking for possible codepaths where this could happen, and haven't surfaced anything yet. I have some further ideas for determining if this is the case, which I'll cover below.
• The last possibility is that this could happen if the "internet connection fails when the notification is being retrieved, or possibly some ultra power saving modes on some devices will let the GCM signal through, but Chrome won't be able to make normal network requests to our server to retrieve the notification due to the ultra power saving mode" - unfortunately, if this is the case, there's not much we can do about this, but the source for this even being a possible failure mode is somewhat tentative and sketchy (ref), so it's hard to know how much weight to give to this scenario.

In order to determine whether the issue is due to a bug in Appboy's service worker, I've released version 1.6.3 of the web sdk which includes increased logging in the service worker - could you:

• upgrade to 1.6.3 of the web sdk
  • https://js.appboycdn.com/web-sdk/1.6/appboy.min.js
  • https://js.appboycdn.com/web-sdk/1.6/appboy.min.css
  • https://js.appboycdn.com/web-sdk/1.6/service-worker.js
• open chrome://serviceworker-internals/
• with that page already open (you can't open it afterwards, or the logs won't show up - it turns out the internals page needs to be open while the events are happening), reproduce the issue
• In the log section on chrome://serviceworker-internals you should at least see "Appboy: service worker 1.6.3 received push" - hopefully along with some subsequent messages which will help us determine what's happening here.

I'll also reach out to you privately in order to get a good user id for us to look for in our server-side logging.

Sorry that this issue has been so hard to track down, hopefully these steps can help us drive towards resolution.

[froodian]

Looping back around to note that @trevorrjohn and I debugged this issue down to a backend bug on our part and patched it - it should now be resolved globally with no sdk upgrades required. Let me know if anybody else continues to experience this behavior, and thanks for your debugging help @trevorrjohn - closing out this issue.