Deploy Mediator with Firebase Plugin

[esune]

Deploy an updated version of the Mediator with the newly-developed Firebase plugin to support push notifications.

[esune]

For push notifications to work we will need the firebase account configured properly in the Mediator. @cvarjao I believe you mentioned there is a BCGov firebase account that we could/should use for this? Can you please confirm? We can plan next steps based on this.

c.c.: @jamshale if more context/details need to be added.

[esune]

The plugin can be added to the deployment without being turned on - this is the default behaviour. When ready to be turned on, it should be activated by using the environment variable USE_FIREBASE_PLUGIN=true and adding the appropriate envvars for the firebase account settings.

[esune]

The BC Wallet team has a Firebase account that can be used to configure the plugin, @WadeBarnes you can connect with them to obtain them.

[WadeBarnes]

@esune, @jamshale, Should the firebase_push_notifications plugin be part of https://github.com/hyperledger/aries-acapy-plugins?

[jamshale]

@WadeBarnes Yes, I think it should be. I was thinking about moving it but never got around to it.

The only reason to not move it is that it's tightly coupled with the mediator right now. Uses the mediator forward message on the event bus. It could be advanced in the future to listen to different events though and not necessarily be tied to the mediator.

[WadeBarnes]

Related PR; https://github.com/bcgov/openshift-aries-mediator-service/pull/35

[WadeBarnes]

@cvarjao, With whom on the wallet team should I connect regarding the Firebase account information? I have a mediator deployed to dev that is ready for configuration.

[cvarjao]

Please check with @bryce-mcmath

[WadeBarnes]

I ran into a few issues getting the firebase_push_notifications plugin to install and load properly. To install it I had to resort to installing it "editable mode" using; pip install --no-cache-dir -e "git+https://github.com/hyperledger/aries-mediator-service.git@main#egg=firebase_push_notifications&subdirectory=acapy/plugins/firebase_push_notifications"

Once I got past that I ran into the following error when the plugin loads and initializes:

2023-12-19 22:23:12,908 aries_cloudagent.resolver WARNING Ledger is not configured, not loading IndyDIDResolver
2023-12-19 22:23:13,079 aries_cloudagent.commands.start ERROR Exception during startup:
Traceback (most recent call last):
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 72, in init
    await startup
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/commands/start.py", line 28, in start_app
    await conductor.setup()
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 122, in setup
    context = await self.context_builder.build_context()
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/config/default_context.py", line 65, in build_context
    await self.load_plugins(context)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/config/default_context.py", line 147, in load_plugins
    await plugin_registry.init_context(context)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/plugin_registry.py", line 211, in init_context
    self.register_protocol_events(context)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/plugin_registry.py", line 305, in register_protocol_events
    mod = ClassLoader.load_module(
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/utils/classloader.py", line 70, in load_module
    return import_module(mod_path, package)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/aries/src/firebase-push-notifications/acapy/plugins/firebase_push_notifications/firebase_push_notifications/v1_0/routes.py", line 16, in <module>
    from .manager import save_device_token, send_message
  File "/home/aries/src/firebase-push-notifications/acapy/plugins/firebase_push_notifications/firebase_push_notifications/v1_0/manager.py", line 12, in <module>
    from .models import FirebaseConnectionRecord
  File "/home/aries/src/firebase-push-notifications/acapy/plugins/firebase_push_notifications/firebase_push_notifications/v1_0/models.py", line 5, in <module>
    from aries_cloudagent.messaging.valid import INDY_ISO8601_DATETIME
ImportError: cannot import name 'INDY_ISO8601_DATETIME' from 'aries_cloudagent.messaging.valid' (/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/messaging/valid.py)

Shutting down

The Dockerfile for the container looks like this:

FROM ghcr.io/hyperledger/aries-cloudagent-python:py3.9-0.10.4

# ========================================================================================================================================
# Install Plugins
# ----------------------------------------------------------------------------------------------------------------------------------------
# Redis PQ
RUN pip install --no-cache-dir git+https://github.com/hyperledger/aries-acapy-plugins.git@main#subdirectory=redis_events
# Firebase Push Notifications
RUN pip install --no-cache-dir -e "git+https://github.com/hyperledger/aries-mediator-service.git@main#egg=firebase_push_notifications&subdirectory=acapy/plugins/firebase_push_notifications"
# ========================================================================================================================================

cc @jamshale

[WadeBarnes]

Looks like INDY_ISO8601_DATETIME is now INDY_ISO8601_DATETIME_VALIDATE.

[jamshale]

@WadeBarnes Ok. Wasn't expecting any of these problems. Changing that import should be fine.

Do you want me to go over it? I hadn't tested anything since the acapy upgrades. There was some unit tests that i think would have failed but their not being run automatically.

[WadeBarnes]

@bryce-mcmath, was going to have a look. I'll see where things are at in the morning and dig in again as needed. I'll let you two coordinate for now. Thanks both

[jamshale]

@WadeBarnes There's a fix for the import error merged to main.

[WadeBarnes]

@jamshale, @bryce-mcmath, Thanks for the fix. I've rebuilt and redeployed the code into dev with both the redis and firebase plugins loaded.

@bryce-mcmath, Are you able to take it for a test drive?

• Mediator URLs here.

[bryce-mcmath]

@WadeBarnes Tested, seems to be working well. There is one minor change I might make to a constant in the firebase plugin but overall it's working well

[WadeBarnes]

@WadeBarnes Tested, seems to be working well. There is one minor change I might make to a constant in the firebase plugin but overall it's working well

Make the change and then I can redeploy for a smoke test before promoting to test.

[bryce-mcmath]

PR here: https://github.com/hyperledger/aries-mediator-service/pull/116

[WadeBarnes]

PR here: hyperledger/aries-mediator-service#116

The dev environment has been updated with the new code.

For future reference, we could update the deployment configuration to mount constants.py from a config map, so we can treat any changes to the file as a config change rather than a code change..

[WadeBarnes]

@bryce-mcmath, Would you deem this ready to go to test?

[bryce-mcmath]

@WadeBarnes yes, good to deploy to test

[WadeBarnes]

@bryce-mcmath, The changes have been promoted to the test environment.

[WadeBarnes]

@bryce-mcmath, How's the testing coming along in the test environment?

[bryce-mcmath]

@WadeBarnes haven't done much testing yet, but we will this week

[WadeBarnes]

@jamshale, @bryce-mcmath, The Firebase plugin is firing off Error messages when non-mobile agents are using it to relay messages. With the load testing taking place on the Traction sandbox and test environments the volumes have been quite large.

Is this message appropriate at this level (error)? Is there a way to better detect whether or not the push notification should be used? Do we need separate controls over the Firebase logging so we can suppress these messages?

[jamshale]

@WadeBarnes I think when I did this I was thinking that if the firebase plugin was enabled that it was expected most agents would be sending tokens and using firebase, and that it might be useful for seeing if the mediator wasn't getting tokens from some mobile agents. This log could be changed to a trace level or even removed completely. There's other ways like the firebase console or looking back at the received token log message to determine the same thing.

[WadeBarnes]

@jamshale, Great, I think reducing it to debug level would be sufficient.

[jleach]

Here is a larger image from the Kibana screenshot:

[jamshale]

I'll change this. Just need to set up an environment and test it.

[jamshale]

PR here. https://github.com/hyperledger/aries-mediator-service/pull/120. I refactored a bit and have only tested with a unit test. The environment setup is quite a lot.

@bryce-mcmath could you test this for me? You should just be able to connect a new bc wallet with firebase disabled on the mobile side.

I will try and get back to this and setup the full env later.

[jamshale]

I tested the changes manually and think they can be merged and deployed whenever we want.

[WadeBarnes]

I tested the changes manually and think they can be merged and deployed whenever we want.

I'll do a build and deploy off the PR so we can test things out before the merge.

[WadeBarnes]

The updated code has been build and deployed to dev. @swcurran, would you be able to run one of your load tests for a short time to generate some traffic please.

[WadeBarnes]

@swcurran, ran a small load test and I confirmed log messages are being generated at the desired log levels.

[WadeBarnes]

I've rebuilt and deployed that latest changes to dev and test following the merge of PR https://github.com/hyperledger/aries-mediator-service/pull/120.

cc @bryce-mcmath, @jamshale, @swcurran, @jleach

[WadeBarnes]

Please reassign to me once this is ready to move forward.

[bryce-mcmath]

@WadeBarnes tested thoroughly in test, wallet team is good to move forward

[WadeBarnes]

Firebase support has been promoted to prod.

cc @bryce-mcmath, @jamshale, @cvarjao, @jleach

[WadeBarnes]

Assigned to @bryce-mcmath for final testing. Please close when successfully completed.

[WadeBarnes]

@bryce-mcmath, How has the testing been going?

[bryce-mcmath]

@WadeBarnes individual tests have been good, no issues, but we haven't yet switched the push notifications feature on in testflight or a release version yet. I'll let you know once we do

[bryce-mcmath]

@WadeBarnes All Testflight testing has gone well and we are about to do a release that includes GA push notifications

[cvarjao]

can we close this one?

[bryce-mcmath]

I believe so. It's been in prod for awhile now without issue