integrate with GitHub for Requests project management

[josh-chamberlain]

Context

We'll use github to communicate about requests. This integration allows us to create issues, and read their status from the app.

• here's the model project.
  • it has some issues in it, all manually-generated. Some of them won't have all the metadata attached, but the project-level custom fields are the target.
• requires #256
• requires https://github.com/Police-Data-Accessibility-Project/data-sources-app/issues/422
• related to https://github.com/Police-Data-Accessibility-Project/data-projects/issues/5

Requirements

• 422

• [ ] #464
• [ ] #465
• [ ] when a request hits the ready to start status, create a new GitHub issue in the project above, in the data-projects repo
• [ ] GitHub is then the source of truth for request_status; infer this from GitHub by checking for it periodically
• [ ] when the page is loaded, limited to once every minute—this may result in fewer refreshes overall)
• [ ] work_required is converted to labels on the issue; record_types_required can also be labels.
  • we should sync these back to our database too, for portability
• [ ] add an issue template to the repo that tells people they should not make a request by making an issue, and points to the intake form

The GitHub Discussions option

• we could use github discussions to prevent new requests from being created except through the app, but this seems complicated
• discussions are kind of higher level research questions, and issues are tasks within them. discussions don't have project management. if we use discussions, we need an orderly way to extract the requests from them—especially for simple cases where 1 request → 1 discussion → 1 task → response, we want to cut the noise a bit.

[maxachis]

* maybe they go into a private repo at first, and get moved into a public repo when they're ready for volunteers

One option is to utilize GitHub discussions, which would enable us to review and discuss data requests before converting them to an issue (which can be easily done using Discussions functionality). GitHub discussions, as I understand it, is tied to the visibility of the repository it is connected with.

Still, having them exist as discussions allows us to refine and potentially talk more with the request provider before converting it to a distinct issue, and avoids having to maintain two separate repositories. We can set it so that users with read access can be disallowed from creating discussions.

At the very least, I think it's worth playing around with to see whether it can work for our purposes.

[josh-chamberlain]

@maxachis I like this idea. In particular, blocking people from creating a new discussion except through the form, which we can't do with issues. It's set up to encourage community involvement and more appropriate from an abstract terms/functions perspective. Either way, I set up discussions for the repo to give it a fighting chance.

misgivings:

• we don't really need to silo these any more, because they're already in their own repo + project
• the discussion doesn't convert to an issue, it creates one—then there are two conversations to track. we could automate this problem away, maybe.
• it'd be nice to talk to the user before we make an official issue out of it; but a discussion is still not private. I don't like my private repo idea. i think a retool middle layer for processing requests could make sense here the same way it does for approving data source submissions.

[maxachis]

@josh-chamberlain Here is the MVP workflow as I am envisioning it (from the backend's perspective). We can expand on this, but I want to make sure I've got the core functionality down.

1. The POST endpoint for /data-requests is called, creating the data request in the backend
2. At some later point, the POST endpoint for /data-requests/github-issue/{ID} is called, which takes the data request as it exists in the database and creates an issue in the Data Projects repository using that data. The github_issue_url field is updated in the database, and possibly the request_status.

Does this align with what you're thinking?

[josh-chamberlain]

@maxachis sorry, I never responded! Yes, this is what I'm thinking. I made some changes to the issue yesterday.

[maxachis]

Github GraphQL API

Relevant Links

• Main API Docs Page
• Using the API to Manage Projects
• Forming calls with GraphQL
• Managing your personal access token
• About Creating Github Apps
• Authenticating as a Github App Installation - "You can make your GitHub App authenticate as an installation in order to make API requests that affect resources owned by the account where the app is installed."
• Using webhooks with Github Apps - we may want to use webhooks for notifying us of changes to the project (or for providing information about changes to the project).
• Building a Github app that responds to Webhook events.

Other notes:

• The project ID for Data Requests is 26

Authenticating

• We will likely need a personal access token with permissions to read and edit this project (or just projects in general, if we can't get that granular).
• Alternatively, we may want to set up a Github App, which appears to be what GH recommends for "Long-lived integrations"

[maxachis]

@josh-chamberlain Alright, after doing some snurfling, here's what I've determined as far as the implementation goes.

I'll also re: @joshuagraber here, so he can sanity check my thinking on this.

Implementation

This feature, as currently designed would involve multiple components:

1. Modifying the project automations (i.e., in https://github.com/orgs/Police-Data-Accessibility-Project/projects/26/workflows ) so that, when an issue is added to the repository, it is automatically added to the project and given a specific status
2. Setting up a /github/data-requests/issues endpoint, whose POST functionality adds an issue to GitHub with the given information,
   1. This requires a GraphQL call
3. Setting up a /github/data-requests/synchronize endpoint, whose POST functionality
   1. Uses a GraphQL call to get all issues (and their statuses) in the repository
   2. ~Checks to see if they don't exist in the Database, and adds them to the database if not~ ignores all issues which do not have a corresponding data request in the database (this should not occur, anyway).
   3. Checks to see if their status is the same as it is in the database, and updates the database status if not
4. Creating an automation manager job which calls /github/data/requests/synchronize on a periodic basis

What are the unknowns?

• At this point, not too much. I've figured out the requisite GraphQL calls I'd need to perform, and I'm already well versed in making endpoints and updating the database.
• There is an unknown in terms of how to operate the GraphQL commands programmatically. Figuring out authentication and getting all of that to work properly would take some additional noodling.
• There is also an unknown (though a minor one) of getting the proper authentication/access information to be able to read information from the project and write new issues to the repository.

This seems convoluted: Is there any other option?

Not as far as I can tell if we want to use GitHub projects, I'm afraid. There are no GitHub Action triggers (https://github.com/orgs/community/discussions/40848) for when a Project's status is changed, nor any Webhooks that do the same, so our only option is to repeatedly poll for updates and modify them in the database as they occur.

However, if we were instead interested in using the future Retool option to manage data requests, that would (in theory) provide us with a means for monitoring and manipulating data request statuses without having to use the clunky synchronization option. However, if we did that, we would lose the tight GitHub integration.

[josh-chamberlain]

@maxachis I don't think we should add issues from github as PDAP requests—syncing status makes sense, but creating requests from our app should be the one and only workflow (at least for now) for simplicity and clarity. We'd only be polling GitHub for status on occasion; we could do this on page view and/or periodically.

[maxachis]

@maxachis I don't think we should add issues from github as PDAP requests—syncing status makes sense, but creating requests from our app should be the one and only workflow (at least for now) for simplicity and clarity. We'd only be polling GitHub for status on occasion; we could do this on page view and/or periodically.

@josh-chamberlain Got it! I've updated the comment accordingly!