Create automation job for sending notifications to users about followed locations

[maxachis]

Follows :

• 478

part of: #26

To implement the above endpoint properly, an automation job will need set up which can take subscribed users and send them notifications about updated data requests and sources.

This requires several other components:

1. A working and long-term v2 database (as of this writing, the only v2 databases are refreshed from v1 production daily)
2. Obviously, this automation job will do nothing if there are no users following anything.

Other things to consider:

• This notification job is designed to run once a month, so it would be a while between setup and when we could actually see it play out as expected in a production environment.
• Hence, it'll probably be a good idea to create test cases in a beta v2 environment, where we:
  • Have one of us follow several locations
  • Add real or test data requests and/or data sources to those locations
  • Manipulate the database entries of these events so that they are the month prior
  • Run the endpoint to confirm that emails are sent.

Requirements

• [x] Develop notifications repository
• [x] Develop notifications automation job
• [ ] Test functionality by adding some notification-worthy events and running the automation job to see that everything works as expected

[josh-chamberlain]

The only thing keeping us from a proper v2 beta is #32, IMO.

[maxachis]

So now with a semi-stable v2 Production Database set up, we can make more progress on this.

Steps I'll need to take to finish setting up:

• [x] Create a user in the database with the NOTIFICATIONS permission (and probably have that be created in the migration script).
• [x] Set up automation job that takes that user's credentials and uses it to call the /notifications endpoint. We can trigger that manually now and then later rejigger it to occur on a regular basis.
• [ ] Then, of course, test it.

[maxachis]

Automation job set up! I've created an automation repository for storing this (and likely eventually other) automation jobs that don't require a whole repository.

https://github.com/Police-Data-Accessibility-Project/automation/tree/main

In automation.pdap.io, I set it to run on the second day of every month.

I gave it a run and it successfully ran to completion, but of course it had no notification jobs to send out. So next step will be to give it some notification jobs, along with some follows, and then test that.

[josh-chamberlain]

@maxachis this is OK—but I'm curious why it wouldn't go in the main repo of the app? It occurs to me that if we changed something about the API which affected notifications, we'd have to make an accompanying PR in that other repository.

[maxachis]

@josh-chamberlain It's a worthwhile question, and one that I suppose depends on our design philosophy!

There's definite benefits in having everything in one big mondo app, in terms of everything being centralized and not having to make parallel PRs.

The downsides, as I see it, are as follows

1. The more stuff you have in a repo, the more stuff needs to be downloaded for that repo. That adds time and resources for every user, every Github Action, every automation job that makes use of it.
2. Organization becomes more fraught -- you're dealing with multiple concerns within the same repository, each with their own documentation needs and distinct dependencies (some of which may conflict). Issues all exist within the same repository. Git commits now have to take into account more potential areas where branches may diverge.
3. Finally, where do we draw the boundary line? The Prod to Dev repository, the Data Source Mirror, the health monitor, these could all be candidates for being included. I think the more we include, the more disadvantages come with having a mondo repository.

Now my views on this have evolved over time -- some of the repositories I mentioned above, like the Data Source Mirror and Health Monitor, could probably be moved to the automation repository just to keep things focused. So I do think some consolidation is in order. But I think too much consolidation risks making a big blob of confusion.

I'll pull @joshuagraber into this, because he may have some insight that maybe we haven't yet considered.

[josh-chamberlain]

@maxachis

1. true
2. true, but I think we could handle it
3. those examples all feel external to me; they're other symbiotic organisms. notifications feels like part of the core service. so, to answer your question...by "feel". hm.

I would also say that consolidating based on automation is less worthwhile, because the only thing they have in common is the fact that they are automated; I don't think it has much effect either way. In fact, maybe my issue is actually calling the repo automations when it doesn't contain all of them (or need to).

So what do you think about, instead, just naming that new repo notifications and leaving them where they are? They are pretty important for us, and will likely grow in scope.

[joshuagraber]

The more stuff you have in a repo, the more stuff needs to be downloaded for that repo. That adds time and resources for every user, every Github Action, every automation job that makes use of it.

Agreed here. But the current contents is not going to break anyone's bandwidth.

I would also say that consolidating based on automation is less worthwhile, because the only thing they have in common is the fact that they are automated; I don't think it has much effect either way. In fact, maybe my issue is actually calling the repo automations when it doesn't contain all of them (or need to).

Agreed here, too. If we're centralizing and sharing automation jobs, I would suggest creating org-level Github Actions and then accessing them via matrices in the GH workflow files that are stored in various repositories. I've been considering doing this for the client workflows that are more or less duplicated in design-system and data-sources-app / ds-v2. If that's a route we're considering, I'm happy to implement it for those shared client workflows as a POC.

So what do you think about, instead, just naming that new repo notifications and leaving them where they are? They are pretty important for us, and will likely grow in scope.

But also agreed here, and I think in this particular case, we can consider notifications a micro-service, and give it its own repo.

[maxachis]

@josh-chamberlain @joshuagraber Alright! The repository has been renamed to notifications!