GitHub integration

[buckyroberts]

Create a periodic task using the Celery Beat scheduler that runs every 5 minutes. This will poll the GitHub API to fetch any PRs that have been merged into any of these repos: https://github.com/thenewboston-developers

For each PR, we will use an LLM to determine the value of the PR. Note that you do not have to write this part. I am currently working on it. For development you can just generate a random number from 100-1,000 instead.

Once the value has been determined, we will create a contribution to reward the contributor with that many coins. This will also increase the amount in the TNB wallet:

• https://github.com/thenewboston-developers/thenewboston-Backend/blob/master/thenewboston/contributions/models/contribution.py
• https://github.com/thenewboston-developers/thenewboston-Backend/blob/master/thenewboston/wallets/models/wallet.py

Our GitHubUser model will be used as the link between GitHub user and thenewboston user. When a GitHub PR is submitted and value assessed, we will look for a GitHubUser with the matching GitHubUser.github_id (GitHub user ID) and reward the GitHubUser.reward_recipient (thenewboston user) that amount. Later we will develop a system to update the GitHubUser dynamically, however for now we can just manually create them for development.

From a user's point of view, this will allow developers to work on the project and be rewarded for their contributions.

[dmugtasimov]

This is comment will collect the task break down (updatable):

1. [DONE] Make a test Github API call
2. [DONE] Add Celery Beat
3. [DONE] Implement Github client (based on https://github.com/PyGithub/PyGithub )
4. [DONE] Add git hub token setting
5. [CANCELED] Refine required scopes for GitHub API access token
6. [CANCELED] Add description of how to acquire GitHub API access token
7. [DONE] Add repo polling
8. [DONE] Implement creation of contribution in case of detecting a new PR
9. [DONE] Implement increasing wallet balance in case of detecting a new PR
10. [DONE] Unitest the implementation

[dmugtasimov]

@buckyroberts I will implement the github integration with Personal access token to for (fine-grained or classic if it works). But there is an option to implement it as GitHub App. I do not know if there any benefits of that in this particular case. Please, let me know if you want me to explore that path

[dmugtasimov]

@buckyroberts There is already a model thenewboston.github.models.repo.Repo. How is it populated? I can get a list of repos from GET https://api.github.com/orgs/thenewboston-developers/repos API call. Should add Repo instances from the API response? Or should I assume Repo prepopulated (I do not think so, since there just name attribute and I am not sure how well "machine readable" it is). Or maybe you do not expect to interact with persisted repos as a part of this task?

[dmugtasimov]

@buckyroberts There is also pull_request webhook available https://docs.github.com/en/webhooks/webhook-events-and-payloads#pull_request . One drawback of the webhooks is that in a rare case they can be lost due network or temporary service unavailability issues (therefore we will need to poll anyway, but can do it much less frequently) and they are harder to implement than active polling, but they allow to get events earlier and reduce network load. Please, let me know it you want me to explore that path

[dmugtasimov]

Pull.issue_id will be renamed to Pull.number, because id has its own meaning in GitHub API and to better mimic GibHub API structure and field naming.

It is inferred from production data that by issue_id the PR number is actually meant (by the magnitude of the value used):

thenewboston=# select * from github_pull;
 id |         created_date          |        modified_date         | issue_id |       title        | repo_id 
----+-------------------------------+------------------------------+----------+--------------------+---------
  1 | 2024-01-16 00:30:54.206773+00 | 2024-01-16 00:30:54.20679+00 |       94 | 91 - Notifications |       1
(1 row)

GET https://api.github.com/repos/thenewboston-developers/Core/pulls?state=all&per_page=1

[
    {
        "url": "https://api.github.com/repos/thenewboston-developers/Core/pulls/151",
        "id": 1224042506,
        "node_id": "PR_kwDOHNY-X85I9WgK",
        "html_url": "https://github.com/thenewboston-developers/Core/pull/151",
        "diff_url": "https://github.com/thenewboston-developers/Core/pull/151.diff",
        "patch_url": "https://github.com/thenewboston-developers/Core/pull/151.patch",
        "issue_url": "https://api.github.com/repos/thenewboston-developers/Core/issues/151",
        "number": 151,
        "state": "closed",
        "locked": false,
        "title": "poetry and pip upgrade",
        "user": {
            "login": "dmugtasimov",
            "id": 749833,
            "node_id": "MDQ6VXNlcjc0OTgzMw==",
            "avatar_url": "https://avatars.githubusercontent.com/u/749833?v=4",
            "gravatar_id": "",
            "url": "https://api.github.com/users/dmugtasimov",
            "html_url": "https://github.com/dmugtasimov",
            "followers_url": "https://api.github.com/users/dmugtasimov/followers",
            "following_url": "https://api.github.com/users/dmugtasimov/following{/other_user}",
            "gists_url": "https://api.github.com/users/dmugtasimov/gists{/gist_id}",
            "starred_url": "https://api.github.com/users/dmugtasimov/starred{/owner}{/repo}",
            "subscriptions_url": "https://api.github.com/users/dmugtasimov/subscriptions",
            "organizations_url": "https://api.github.com/users/dmugtasimov/orgs",
            "repos_url": "https://api.github.com/users/dmugtasimov/repos",
            "events_url": "https://api.github.com/users/dmugtasimov/events{/privacy}",
            "received_events_url": "https://api.github.com/users/dmugtasimov/received_events",
            "type": "User",
            "site_admin": false
        },
        "body": null,
        "created_at": "2023-01-31T23:55:32Z",
        "updated_at": "2023-02-01T00:18:40Z",
        "closed_at": "2023-02-01T00:18:40Z",
        "merged_at": "2023-02-01T00:18:40Z",
        "merge_commit_sha": "8e35f74e8b4966339c7f1874224ddc84d2fe1b23",
        "assignee": null,
        "assignees": [
        ],
        "requested_reviewers": [
        ],
        "requested_teams": [
        ],
        "labels": [
        ],
        "milestone": null,
        "draft": false,
        "commits_url": "https://api.github.com/repos/thenewboston-developers/Core/pulls/151/commits",
        "review_comments_url": "https://api.github.com/repos/thenewboston-developers/Core/pulls/151/comments",
        "review_comment_url": "https://api.github.com/repos/thenewboston-developers/Core/pulls/comments{/number}",
        "comments_url": "https://api.github.com/repos/thenewboston-developers/Core/issues/151/comments",
        "statuses_url": "https://api.github.com/repos/thenewboston-developers/Core/statuses/6c13e0960fa3d142ac0808089670a3830536c86d",
        "head": {
            "label": "thenewboston-developers:fix/database-out-of-connections",
            "ref": "fix/database-out-of-connections",
            "sha": "6c13e0960fa3d142ac0808089670a3830536c86d",
            "user": {
                "login": "thenewboston-developers",
                "id": 12706692,
                "node_id": "MDEyOk9yZ2FuaXphdGlvbjEyNzA2Njky",
                "avatar_url": "https://avatars.githubusercontent.com/u/12706692?v=4",
                "gravatar_id": "",
                "url": "https://api.github.com/users/thenewboston-developers",
                "html_url": "https://github.com/thenewboston-developers",
                "followers_url": "https://api.github.com/users/thenewboston-developers/followers",
                "following_url": "https://api.github.com/users/thenewboston-developers/following{/other_user}",
                "gists_url": "https://api.github.com/users/thenewboston-developers/gists{/gist_id}",
                "starred_url": "https://api.github.com/users/thenewboston-developers/starred{/owner}{/repo}",
                "subscriptions_url": "https://api.github.com/users/thenewboston-developers/subscriptions",
                "organizations_url": "https://api.github.com/users/thenewboston-developers/orgs",
                "repos_url": "https://api.github.com/users/thenewboston-developers/repos",
                "events_url": "https://api.github.com/users/thenewboston-developers/events{/privacy}",
                "received_events_url": "https://api.github.com/users/thenewboston-developers/received_events",
                "type": "Organization",
                "site_admin": false
            },
            "repo": {
                "id": 483802719,
                "node_id": "R_kgDOHNY-Xw",
                "name": "Core",
                "full_name": "thenewboston-developers/Core",
                "private": false,
                "owner": {
                    "login": "thenewboston-developers",
                    "id": 12706692,
                    "node_id": "MDEyOk9yZ2FuaXphdGlvbjEyNzA2Njky",
                    "avatar_url": "https://avatars.githubusercontent.com/u/12706692?v=4",
                    "gravatar_id": "",
                    "url": "https://api.github.com/users/thenewboston-developers",
                    "html_url": "https://github.com/thenewboston-developers",
                    "followers_url": "https://api.github.com/users/thenewboston-developers/followers",
                    "following_url": "https://api.github.com/users/thenewboston-developers/following{/other_user}",
                    "gists_url": "https://api.github.com/users/thenewboston-developers/gists{/gist_id}",
                    "starred_url": "https://api.github.com/users/thenewboston-developers/starred{/owner}{/repo}",
                    "subscriptions_url": "https://api.github.com/users/thenewboston-developers/subscriptions",
                    "organizations_url": "https://api.github.com/users/thenewboston-developers/orgs",
                    "repos_url": "https://api.github.com/users/thenewboston-developers/repos",
                    "events_url": "https://api.github.com/users/thenewboston-developers/events{/privacy}",
                    "received_events_url": "https://api.github.com/users/thenewboston-developers/received_events",
                    "type": "Organization",
                    "site_admin": false
                },
                "html_url": "https://github.com/thenewboston-developers/Core",
                "description": "Core messaging server.",
                "fork": false,
                "url": "https://api.github.com/repos/thenewboston-developers/Core",
                "forks_url": "https://api.github.com/repos/thenewboston-developers/Core/forks",
                "keys_url": "https://api.github.com/repos/thenewboston-developers/Core/keys{/key_id}",
                "collaborators_url": "https://api.github.com/repos/thenewboston-developers/Core/collaborators{/collaborator}",
                "teams_url": "https://api.github.com/repos/thenewboston-developers/Core/teams",
                "hooks_url": "https://api.github.com/repos/thenewboston-developers/Core/hooks",
                "issue_events_url": "https://api.github.com/repos/thenewboston-developers/Core/issues/events{/number}",
                "events_url": "https://api.github.com/repos/thenewboston-developers/Core/events",
                "assignees_url": "https://api.github.com/repos/thenewboston-developers/Core/assignees{/user}",
                "branches_url": "https://api.github.com/repos/thenewboston-developers/Core/branches{/branch}",
                "tags_url": "https://api.github.com/repos/thenewboston-developers/Core/tags",
                "blobs_url": "https://api.github.com/repos/thenewboston-developers/Core/git/blobs{/sha}",
                "git_tags_url": "https://api.github.com/repos/thenewboston-developers/Core/git/tags{/sha}",
                "git_refs_url": "https://api.github.com/repos/thenewboston-developers/Core/git/refs{/sha}",
                "trees_url": "https://api.github.com/repos/thenewboston-developers/Core/git/trees{/sha}",
                "statuses_url": "https://api.github.com/repos/thenewboston-developers/Core/statuses/{sha}",
                "languages_url": "https://api.github.com/repos/thenewboston-developers/Core/languages",
                "stargazers_url": "https://api.github.com/repos/thenewboston-developers/Core/stargazers",
                "contributors_url": "https://api.github.com/repos/thenewboston-developers/Core/contributors",
                "subscribers_url": "https://api.github.com/repos/thenewboston-developers/Core/subscribers",
                "subscription_url": "https://api.github.com/repos/thenewboston-developers/Core/subscription",
                "commits_url": "https://api.github.com/repos/thenewboston-developers/Core/commits{/sha}",
                "git_commits_url": "https://api.github.com/repos/thenewboston-developers/Core/git/commits{/sha}",
                "comments_url": "https://api.github.com/repos/thenewboston-developers/Core/comments{/number}",
                "issue_comment_url": "https://api.github.com/repos/thenewboston-developers/Core/issues/comments{/number}",
                "contents_url": "https://api.github.com/repos/thenewboston-developers/Core/contents/{+path}",
                "compare_url": "https://api.github.com/repos/thenewboston-developers/Core/compare/{base}...{head}",
                "merges_url": "https://api.github.com/repos/thenewboston-developers/Core/merges",
                "archive_url": "https://api.github.com/repos/thenewboston-developers/Core/{archive_format}{/ref}",
                "downloads_url": "https://api.github.com/repos/thenewboston-developers/Core/downloads",
                "issues_url": "https://api.github.com/repos/thenewboston-developers/Core/issues{/number}",
                "pulls_url": "https://api.github.com/repos/thenewboston-developers/Core/pulls{/number}",
                "milestones_url": "https://api.github.com/repos/thenewboston-developers/Core/milestones{/number}",
                "notifications_url": "https://api.github.com/repos/thenewboston-developers/Core/notifications{?since,all,participating}",
                "labels_url": "https://api.github.com/repos/thenewboston-developers/Core/labels{/name}",
                "releases_url": "https://api.github.com/repos/thenewboston-developers/Core/releases{/id}",
                "deployments_url": "https://api.github.com/repos/thenewboston-developers/Core/deployments",
                "created_at": "2022-04-20T20:25:12Z",
                "updated_at": "2024-02-22T16:54:16Z",
                "pushed_at": "2023-02-01T00:18:40Z",
                "git_url": "git://github.com/thenewboston-developers/Core.git",
                "ssh_url": "git@github.com:thenewboston-developers/Core.git",
                "clone_url": "https://github.com/thenewboston-developers/Core.git",
                "svn_url": "https://github.com/thenewboston-developers/Core",
                "homepage": null,
                "size": 473,
                "stargazers_count": 29,
                "watchers_count": 29,
                "language": "Python",
                "has_issues": true,
                "has_projects": true,
                "has_downloads": true,
                "has_wiki": true,
                "has_pages": false,
                "has_discussions": false,
                "forks_count": 13,
                "mirror_url": null,
                "archived": false,
                "disabled": false,
                "open_issues_count": 10,
                "license": {
                    "key": "mit",
                    "name": "MIT License",
                    "spdx_id": "MIT",
                    "url": "https://api.github.com/licenses/mit",
                    "node_id": "MDc6TGljZW5zZTEz"
                },
                "allow_forking": true,
                "is_template": false,
                "web_commit_signoff_required": false,
                "topics": [
                ],
                "visibility": "public",
                "forks": 13,
                "open_issues": 10,
                "watchers": 29,
                "default_branch": "master"
            }
        },
        "base": {
            "label": "thenewboston-developers:master",
            "ref": "master",
            "sha": "d8c5a825fadbbc941414f081cdadf5aac4df500f",
            "user": {
                "login": "thenewboston-developers",
                "id": 12706692,
                "node_id": "MDEyOk9yZ2FuaXphdGlvbjEyNzA2Njky",
                "avatar_url": "https://avatars.githubusercontent.com/u/12706692?v=4",
                "gravatar_id": "",
                "url": "https://api.github.com/users/thenewboston-developers",
                "html_url": "https://github.com/thenewboston-developers",
                "followers_url": "https://api.github.com/users/thenewboston-developers/followers",
                "following_url": "https://api.github.com/users/thenewboston-developers/following{/other_user}",
                "gists_url": "https://api.github.com/users/thenewboston-developers/gists{/gist_id}",
                "starred_url": "https://api.github.com/users/thenewboston-developers/starred{/owner}{/repo}",
                "subscriptions_url": "https://api.github.com/users/thenewboston-developers/subscriptions",
                "organizations_url": "https://api.github.com/users/thenewboston-developers/orgs",
                "repos_url": "https://api.github.com/users/thenewboston-developers/repos",
                "events_url": "https://api.github.com/users/thenewboston-developers/events{/privacy}",
                "received_events_url": "https://api.github.com/users/thenewboston-developers/received_events",
                "type": "Organization",
                "site_admin": false
            },
            "repo": {
                "id": 483802719,
                "node_id": "R_kgDOHNY-Xw",
                "name": "Core",
                "full_name": "thenewboston-developers/Core",
                "private": false,
                "owner": {
                    "login": "thenewboston-developers",
                    "id": 12706692,
                    "node_id": "MDEyOk9yZ2FuaXphdGlvbjEyNzA2Njky",
                    "avatar_url": "https://avatars.githubusercontent.com/u/12706692?v=4",
                    "gravatar_id": "",
                    "url": "https://api.github.com/users/thenewboston-developers",
                    "html_url": "https://github.com/thenewboston-developers",
                    "followers_url": "https://api.github.com/users/thenewboston-developers/followers",
                    "following_url": "https://api.github.com/users/thenewboston-developers/following{/other_user}",
                    "gists_url": "https://api.github.com/users/thenewboston-developers/gists{/gist_id}",
                    "starred_url": "https://api.github.com/users/thenewboston-developers/starred{/owner}{/repo}",
                    "subscriptions_url": "https://api.github.com/users/thenewboston-developers/subscriptions",
                    "organizations_url": "https://api.github.com/users/thenewboston-developers/orgs",
                    "repos_url": "https://api.github.com/users/thenewboston-developers/repos",
                    "events_url": "https://api.github.com/users/thenewboston-developers/events{/privacy}",
                    "received_events_url": "https://api.github.com/users/thenewboston-developers/received_events",
                    "type": "Organization",
                    "site_admin": false
                },
                "html_url": "https://github.com/thenewboston-developers/Core",
                "description": "Core messaging server.",
                "fork": false,
                "url": "https://api.github.com/repos/thenewboston-developers/Core",
                "forks_url": "https://api.github.com/repos/thenewboston-developers/Core/forks",
                "keys_url": "https://api.github.com/repos/thenewboston-developers/Core/keys{/key_id}",
                "collaborators_url": "https://api.github.com/repos/thenewboston-developers/Core/collaborators{/collaborator}",
                "teams_url": "https://api.github.com/repos/thenewboston-developers/Core/teams",
                "hooks_url": "https://api.github.com/repos/thenewboston-developers/Core/hooks",
                "issue_events_url": "https://api.github.com/repos/thenewboston-developers/Core/issues/events{/number}",
                "events_url": "https://api.github.com/repos/thenewboston-developers/Core/events",
                "assignees_url": "https://api.github.com/repos/thenewboston-developers/Core/assignees{/user}",
                "branches_url": "https://api.github.com/repos/thenewboston-developers/Core/branches{/branch}",
                "tags_url": "https://api.github.com/repos/thenewboston-developers/Core/tags",
                "blobs_url": "https://api.github.com/repos/thenewboston-developers/Core/git/blobs{/sha}",
                "git_tags_url": "https://api.github.com/repos/thenewboston-developers/Core/git/tags{/sha}",
                "git_refs_url": "https://api.github.com/repos/thenewboston-developers/Core/git/refs{/sha}",
                "trees_url": "https://api.github.com/repos/thenewboston-developers/Core/git/trees{/sha}",
                "statuses_url": "https://api.github.com/repos/thenewboston-developers/Core/statuses/{sha}",
                "languages_url": "https://api.github.com/repos/thenewboston-developers/Core/languages",
                "stargazers_url": "https://api.github.com/repos/thenewboston-developers/Core/stargazers",
                "contributors_url": "https://api.github.com/repos/thenewboston-developers/Core/contributors",
                "subscribers_url": "https://api.github.com/repos/thenewboston-developers/Core/subscribers",
                "subscription_url": "https://api.github.com/repos/thenewboston-developers/Core/subscription",
                "commits_url": "https://api.github.com/repos/thenewboston-developers/Core/commits{/sha}",
                "git_commits_url": "https://api.github.com/repos/thenewboston-developers/Core/git/commits{/sha}",
                "comments_url": "https://api.github.com/repos/thenewboston-developers/Core/comments{/number}",
                "issue_comment_url": "https://api.github.com/repos/thenewboston-developers/Core/issues/comments{/number}",
                "contents_url": "https://api.github.com/repos/thenewboston-developers/Core/contents/{+path}",
                "compare_url": "https://api.github.com/repos/thenewboston-developers/Core/compare/{base}...{head}",
                "merges_url": "https://api.github.com/repos/thenewboston-developers/Core/merges",
                "archive_url": "https://api.github.com/repos/thenewboston-developers/Core/{archive_format}{/ref}",
                "downloads_url": "https://api.github.com/repos/thenewboston-developers/Core/downloads",
                "issues_url": "https://api.github.com/repos/thenewboston-developers/Core/issues{/number}",
                "pulls_url": "https://api.github.com/repos/thenewboston-developers/Core/pulls{/number}",
                "milestones_url": "https://api.github.com/repos/thenewboston-developers/Core/milestones{/number}",
                "notifications_url": "https://api.github.com/repos/thenewboston-developers/Core/notifications{?since,all,participating}",
                "labels_url": "https://api.github.com/repos/thenewboston-developers/Core/labels{/name}",
                "releases_url": "https://api.github.com/repos/thenewboston-developers/Core/releases{/id}",
                "deployments_url": "https://api.github.com/repos/thenewboston-developers/Core/deployments",
                "created_at": "2022-04-20T20:25:12Z",
                "updated_at": "2024-02-22T16:54:16Z",
                "pushed_at": "2023-02-01T00:18:40Z",
                "git_url": "git://github.com/thenewboston-developers/Core.git",
                "ssh_url": "git@github.com:thenewboston-developers/Core.git",
                "clone_url": "https://github.com/thenewboston-developers/Core.git",
                "svn_url": "https://github.com/thenewboston-developers/Core",
                "homepage": null,
                "size": 473,
                "stargazers_count": 29,
                "watchers_count": 29,
                "language": "Python",
                "has_issues": true,
                "has_projects": true,
                "has_downloads": true,
                "has_wiki": true,
                "has_pages": false,
                "has_discussions": false,
                "forks_count": 13,
                "mirror_url": null,
                "archived": false,
                "disabled": false,
                "open_issues_count": 10,
                "license": {
                    "key": "mit",
                    "name": "MIT License",
                    "spdx_id": "MIT",
                    "url": "https://api.github.com/licenses/mit",
                    "node_id": "MDc6TGljZW5zZTEz"
                },
                "allow_forking": true,
                "is_template": false,
                "web_commit_signoff_required": false,
                "topics": [
                ],
                "visibility": "public",
                "forks": 13,
                "open_issues": 10,
                "watchers": 29,
                "default_branch": "master"
            }
        },
        "_links": {
            "self": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/pulls/151"
            },
            "html": {
                "href": "https://github.com/thenewboston-developers/Core/pull/151"
            },
            "issue": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/issues/151"
            },
            "comments": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/issues/151/comments"
            },
            "review_comments": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/pulls/151/comments"
            },
            "review_comment": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/pulls/comments{/number}"
            },
            "commits": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/pulls/151/commits"
            },
            "statuses": {
                "href": "https://api.github.com/repos/thenewboston-developers/Core/statuses/6c13e0960fa3d142ac0808089670a3830536c86d"
            }
        },
        "author_association": "MEMBER",
        "auto_merge": null,
        "active_lock_reason": null
    }
]

Another option is to do something like this (issue_id maps to id in GitHub API pull object, number maps to number):

class Pull(CreatedModified):
    issue_id = models.PositiveIntegerField(unique=True)
    number = models.PositiveIntegerField()
    repo = models.ForeignKey('github.Repo', on_delete=models.CASCADE)
    title = models.CharField(max_length=256)

    class Meta:
        constraints = [
            models.UniqueConstraint(fields=['number', 'repo'], name='unique_number_repo')
        ]

    def __str__(self):
        return f'ID: {self.pk} | Issue ID: {self.issue_id} | Title: {self.title}'

@buckyroberts Please, let me know if you want to keep both issue_id and number

[dmugtasimov]

WIP branch: https://github.com/thenewboston-developers/thenewboston-Backend/tree/108-github-integration

[dmugtasimov]

@buckyroberts please, review https://github.com/thenewboston-developers/thenewboston-Backend/pull/111

Once you approve, please, let me know and I then will merge and deploy , so I can fix issues quickly myself in case there are any.

You are welcome to ask questions about refactorings I made, will be happy to provide my reason where unclear.

I added some libraries and upgraded some others, so you will need to run make update locally.