guardian / mobile-n10n

n10n for nOTIFICATIOn
Apache License 2.0
26 stars 4 forks source link

LIVE-6383 send requests to Firebase API directly #1216

Closed waisingyiu closed 6 months ago

waisingyiu commented 6 months ago

What does this change?

The Firebase cloud message lagacy HTTP endpoint is going to be discontinued this June, and there is no replacement for the multicast API in the HTTP API v1. We have to explore alternatives to improve the throughput of our notification service in order to achieve the 90in2 SLA we are seeing today with the soo-to-be-expired multicast API.

This PR implements direct firebase HTTP calls over multiplexed HTTP/2 protocol. As this Firebase FAQ suggests, 

The HTTP v1 API over HTTP/2 performs similarly for 99.9% of multicast requests (sending < 100 tokens). For outlier use cases (sending 1000 tokens), it achieves up to a third of the throughput rate, so additional concurrency is needed to optimize for this atypical use case.

We are going to validate this option over small notifications and apply it to larger notifications if we have more confidence on its throughput.

We have made the following changes:

  1. implemented client class to directly invoke Firebase's individual send API using JDK's HttpClient which supports multiplexed HTTP/2
  2. changed the FcmClient class to support both the multicast API we are using now as well as the individual send API which we have implemented in this PR
  3. added a configuration item on the topics to which we send notification using the individual send
  4. added a configuration item to control the concurrency when sending via individual send
  5. added test cases

How to test

The changes were tested on CODE and notification can be sent to my simulator. However, we only have a very small number of device token on CODE database (compared to PROD) so we need to run this change as an experiment on PROD.

After the changes are deployed to PROD, we may first enable the individual send on content notification first (by setting tag/ and contributor/ in the selected topics).

When it is shown to send notifications correctly with reasonable throughput, we may extend the topics to breaking sport notification (such as 'breaking/international-sport`) and monitor its throughput all along. As we gain confidence, we may gradually extend it to cover other breaking news topic.

Some parameter tuning may be needed during the experiment.

How can we measure success?

Notifications can be sent with throughput comparable to what we have achieved with multicast API.

Have we considered potential risks?

It may hit some resources problem or have some latency issues. That's the reason we started with small notification.

github-actions[bot] commented 6 months ago

Deploy build 4346 of mobile-n10n:notificationworkerlambda to CODE

All deployment options - [Deploy build 4346 of `mobile-n10n:notificationworkerlambda` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Anotificationworkerlambda&build=4346&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4346 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Anotificationworkerlambda&build=4346&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Anotificationworkerlambda&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4301 of mobile-n10n:eventconsumer to CODE

All deployment options - [Deploy build 4301 of `mobile-n10n:eventconsumer` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Aeventconsumer&build=4301&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4301 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Aeventconsumer&build=4301&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Aeventconsumer&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4304 of mobile-n10n:schedule to CODE

All deployment options - [Deploy build 4304 of `mobile-n10n:schedule` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Aschedule&build=4304&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4304 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Aschedule&build=4304&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Aschedule&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4302 of mobile-n10n:fakebreakingnewslambda to CODE

All deployment options - [Deploy build 4302 of `mobile-n10n:fakebreakingnewslambda` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Afakebreakingnewslambda&build=4302&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4302 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Afakebreakingnewslambda&build=4302&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Afakebreakingnewslambda&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4305 of mobile-n10n:football to CODE

All deployment options - [Deploy build 4305 of `mobile-n10n:football` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Afootball&build=4305&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4305 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Afootball&build=4305&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Afootball&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4302 of mobile-n10n:reportextractor to CODE

All deployment options - [Deploy build 4302 of `mobile-n10n:reportextractor` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Areportextractor&build=4302&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4302 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Areportextractor&build=4302&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Areportextractor&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4301 of mobile-n10n:report to CODE

All deployment options - [Deploy build 4301 of `mobile-n10n:report` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Areport&build=4301&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4301 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Areport&build=4301&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Areport&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4310 of mobile-n10n:notification to CODE

All deployment options - [Deploy build 4310 of `mobile-n10n:notification` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Anotification&build=4310&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4310 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Anotification&build=4310&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Anotification&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4311 of mobile-n10n:slomonitor to CODE

All deployment options - [Deploy build 4311 of `mobile-n10n:slomonitor` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Aslomonitor&build=4311&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4311 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Aslomonitor&build=4311&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Aslomonitor&stage=CODE)

From guardian/actions-riff-raff.

github-actions[bot] commented 6 months ago

Deploy build 4308 of mobile-n10n:registration to CODE

All deployment options - [Deploy build 4308 of `mobile-n10n:registration` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Aregistration&build=4308&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4308 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Aregistration&build=4308&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Aregistration&stage=CODE)

From guardian/actions-riff-raff.