search

guardian / mobile-n10n

n10n for nOTIFICATIOn
Apache License 2.0
26 stars 4 forks source link

LIVE-6468 Reuse http client across invocations #1218

Closed waisingyiu closed 6 months ago

waisingyiu commented 6 months ago

What does this change?

The PR #1217 creates a HttpClient instance for the processing of each SQS client so that we don't send too many requests over the same connection.

However, the AWS lambda service reuses execution environment when it needs to invoke a lambda shortly after one has finished. So it is possible that within the same Java runtime, the AndroidSender instance is used to process SQS messages many times and, as a result, creates too many HttpClient instances and the connections that are left idle. The connections consume the limit on the file descriptors, which is 1024 for Java lambda runtime. It is likely to happen when we send a huge notification (more than 100,000).

What we really want is that we have dedicated HttpClient instances for each SQS message, but we want to reuse these instances in the subsequent invocation if the same runtime environment is reused.

This PR creates 20 HttpClient instances upfront (we process at most 20 SQS messages at a time) and uses them when processing the stream of SQS messages.

This PR also creates a new configuration to control the max connection pool size to avoid "too-many-opened-files" exception.

How to test

We invoked the Android sender locally with two batches of SQS messages (with 3 messages in each batch). We printed the instance in the log message here and we saw from the logs that the three FcmClient instances were reused across invocation:

First batch of SQS messages: Individual send request completed - com.gu.notifications.worker.delivery.fcm.FcmClient@19d807f9

Individual send request completed - com.gu.notifications.worker.delivery.fcm.FcmClient@591354dd

Individual send request completed - com.gu.notifications.worker.delivery.fcm.FcmClient@13fa8ac

Second batch of SQS messages:

Individual send request completed - com.gu.notifications.worker.delivery.fcm.FcmClient@19d807f9

Individual send request completed - com.gu.notifications.worker.delivery.fcm.FcmClient@591354dd

Individual send request completed - com.gu.notifications.worker.delivery.fcm.FcmClient@13fa8ac

The service was also tested on CODE and notification was sent successfully to android app on emulator.

github-actions[bot] commented 6 months ago

Deploy build 4358 of mobile-n10n:notificationworkerlambda to CODE

All deployment options - [Deploy build 4358 of `mobile-n10n:notificationworkerlambda` to CODE](https://riffraff.gutools.co.uk/deployment/deployAgain?project=mobile-n10n%3Anotificationworkerlambda&build=4358&stage=CODE&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 4358 to CODE by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=mobile-n10n%3Anotificationworkerlambda&build=4358&stage=CODE&updateStrategy=MostlyHarmless) - [What's on CODE right now?](https://riffraff.gutools.co.uk/deployment/history?projectName=mobile-n10n%3Anotificationworkerlambda&stage=CODE)

From guardian/actions-riff-raff.