sahsisunny commented 1 year ago

[RFC]: Store Org's all Pull Requests in Real Dev Squad Backend Database and Webhook Integration

Feature Name: Store Org's all Pull Requests in Real Dev Squad Backend Database and Webhook Integration

Summary

This RFC proposes a solution to store all pull requests from the Real Dev Squad organization in the Real Dev Squad backend database. It also suggests integrating a webhook to receive real-time updates from GitHub and keep the backend database synchronized.

Motivation

The purpose of this RFC is to outline a solution for storing Real Dev Squad's GitHub pull requests in the Real Dev Squad backend database. By integrating a webhook, the backend database can be updated in real-time, ensuring data consistency and providing a streamlined workflow.

Detailed design

1. Introduction

This RFC proposes storing the GitHub pull requests of the Real Dev Squad organization in the Real Dev Squad backend database. It also suggests using a webhook to retrieve the latest data from GitHub and keep the backend database updated in real-time.

2. Background

Real Dev Squad uses GitHub for source code management and collaboration. The Real Dev Squad backend database serves as a central repository for various organization data. To streamline workflows and maintain data consistency, it is essential to synchronize pull request information from GitHub into the backend database.

3. Objectives

The objectives of this RFC are as follows:

Store pull request data in the backend database: Extract relevant information from GitHub pull requests and store it in the backend database.
Real-time updates: Implement a webhook to receive notifications from GitHub for pull request creation, updates, and closures, enabling real-time updates in the backend database.
Data integrity: Ensure stored pull request data is accurate, up-to-date, and consistent with corresponding pull requests in GitHub.

4. Proposed Solution

4.1. Database Schema

Design an appropriate database schema to store pull request data in the backend database. The schema should include fields such as pull request ID, title, description, author, creation timestamp, last update timestamp, status (open/closed), and other relevant information.

4.2. API Endpoint

Create an API endpoint in the Real Dev Squad backend to handle incoming requests for storing pull request data. The endpoint should process received data and persist it in the backend database.

4.3. Webhook Integration

Integrate a webhook mechanism with GitHub to receive event notifications related to pull requests. Configure the webhook to send notifications to the designated endpoint in the backend whenever a pull request is created, updated, or closed.

4.4. Event Processing

Upon receiving a webhook notification, the backend should authenticate the request, extract relevant data associated with the pull request, and process it. The processed data should be stored or used to update the corresponding record in the backend database.

4.5. Error Handling

Implement appropriate error handling mechanisms to address cases where webhook notifications fail to be processed or stored in the backend database. This may include logging, error notifications, and retry mechanisms to maintain data consistency.

5. Security Considerations

Ensure the API endpoint and webhook integration are secured using appropriate authentication mechanisms (e.g., OAuth or API keys) to prevent unauthorized access and data tampering.

6. Future Enhancements

In the future, consider additional features to enhance the pull request integration, such as implementing a two-way synchronization to update pull request status in GitHub when modified in the backend database.

Drawbacks

Drawbacks of the proposed design are:

Additional configuration and maintenance requirements for the webhook integration.
The need for additional security measures to protect the API endpoint from unauthorized access and data tampering.
Potential requirement for additional resources to implement and maintain the solution.

Unresolved questions

[ ] Name of the API endpoint to be created in the backend. For example - /pull-requests.
[ ] GitHub API endpoint to be used for retrieving pull request data.
[ ] Database schema for storing pull request data in the backend database.
[ ] How will the API endpoint process incoming requests and store pull request data in the backend database?
[ ] How will the API endpoint handle errors and ensure data consistency?
[ ] How will the webhook be configured to receive notifications from GitHub?
[ ] How will the backend authenticate webhook requests?
[ ] How will the backend process webhook notifications and store pull request data in the backend database?
[ ] What security measures will be implemented to protect the API endpoint from unauthorized access and data tampering?
[ ] Does API have GET, POST, PUT, DELETE methods?

Conclusion

This RFC proposes storing Real Dev Squad's GitHub pull requests in the Real Dev Squad backend database through webhook integration. The solution allows for real-time updates and ensures data consistency. Additional configuration and maintenance efforts may be required for the webhook integration and API endpoint. Security measures must be implemented to protect against unauthorized access and data tampering. Alternatives considered include periodic data storage using scheduled jobs, which do not provide real-time updates.

sahsisunny commented 1 year ago

[
  {
    "url": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/576",
    "id": 1358171923,
    "node_id": "PR_kwDOEQJXR85Q9A8T",
    "html_url": "https://github.com/Real-Dev-Squad/website-status/pull/576",
    "diff_url": "https://github.com/Real-Dev-Squad/website-status/pull/576.diff",
    "patch_url": "https://github.com/Real-Dev-Squad/website-status/pull/576.patch",
    "issue_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues/576",
    "number": 576,
    "state": "closed",
    "locked": false,
    "title": "Create pull_request_template.md",
    "user": {
      "login": "bhtibrewal",
      "id": 42600164,
      "node_id": "MDQ6VXNlcjQyNjAwMTY0",
      "avatar_url": "https://avatars.githubusercontent.com/u/42600164?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/bhtibrewal",
      "html_url": "https://github.com/bhtibrewal",
      "followers_url": "https://api.github.com/users/bhtibrewal/followers",
      "following_url": "https://api.github.com/users/bhtibrewal/following{/other_user}",
      "gists_url": "https://api.github.com/users/bhtibrewal/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/bhtibrewal/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/bhtibrewal/subscriptions",
      "organizations_url": "https://api.github.com/users/bhtibrewal/orgs",
      "repos_url": "https://api.github.com/users/bhtibrewal/repos",
      "events_url": "https://api.github.com/users/bhtibrewal/events{/privacy}",
      "received_events_url": "https://api.github.com/users/bhtibrewal/received_events",
      "type": "User",
      "site_admin": false
    },
    "body": "Issue: #577\r\nadded a template to standardize PRs",
    "created_at": "2023-05-21T10:13:57Z",
    "updated_at": "2023-05-25T16:44:26Z",
    "closed_at": "2023-05-25T16:44:25Z",
    "merged_at": "2023-05-25T16:44:25Z",
    "merge_commit_sha": "4dad25f6558e00cccb12e140b2bb70620095f890",
    "assignee": {
      // Same as user
    },
    "assignees": [
      // Same as user but in array
    ],
    "requested_reviewers": [],
    "requested_teams": [],
    "labels": [],
    "milestone": null,
    "draft": false,
    "commits_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/576/commits",
    "review_comments_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/576/comments",
    "review_comment_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/comments{/number}",
    "comments_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues/576/comments",
    "statuses_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/statuses/087cf71ba3b38237a3e5217a9ab3816a871f3849",
    "head": {
      "label": "Real-Dev-Squad:make-pull_request_template",
      "ref": "make-pull_request_template",
      "sha": "087cf71ba3b38237a3e5217a9ab3816a871f3849",
      "user": {
        // Same as user
      },
      "repo": {
        "id": 285366087,
        "node_id": "MDEwOlJlcG9zaXRvcnkyODUzNjYwODc=",
        "name": "website-status",
        "full_name": "Real-Dev-Squad/website-status",
        "private": false,
        "owner": {
          // Same as user
        },
        "html_url": "https://github.com/Real-Dev-Squad/website-status",
        "description": "Shows a roadmap of the ongoing projects being done",
        "fork": false,
        "url": "https://api.github.com/repos/Real-Dev-Squad/website-status",
        "forks_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/forks",
        "keys_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/keys{/key_id}",
        "collaborators_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/collaborators{/collaborator}",
        "teams_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/teams",
        "hooks_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/hooks",
        "issue_events_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues/events{/number}",
        "events_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/events",
        "assignees_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/assignees{/user}",
        "branches_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/branches{/branch}",
        "tags_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/tags",
        "blobs_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/git/blobs{/sha}",
        "git_tags_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/git/tags{/sha}",
        "git_refs_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/git/refs{/sha}",
        "trees_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/git/trees{/sha}",
        "statuses_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/statuses/{sha}",
        "languages_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/languages",
        "stargazers_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/stargazers",
        "contributors_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/contributors",
        "subscribers_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/subscribers",
        "subscription_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/subscription",
        "commits_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/commits{/sha}",
        "git_commits_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/git/commits{/sha}",
        "comments_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/comments{/number}",
        "issue_comment_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues/comments{/number}",
        "contents_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/contents/{+path}",
        "compare_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/compare/{base}...{head}",
        "merges_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/merges",
        "archive_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/{archive_format}{/ref}",
        "downloads_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/downloads",
        "issues_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues{/number}",
        "pulls_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls{/number}",
        "milestones_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/milestones{/number}",
        "notifications_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/notifications{?since,all,participating}",
        "labels_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/labels{/name}",
        "releases_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/releases{/id}",
        "deployments_url": "https://api.github.com/repos/Real-Dev-Squad/website-status/deployments",
        "created_at": "2020-08-05T17:58:40Z",
        "updated_at": "2023-05-08T10:34:54Z",
        "pushed_at": "2023-05-25T17:21:30Z",
        "git_url": "git://github.com/Real-Dev-Squad/website-status.git",
        "ssh_url": "git@github.com:Real-Dev-Squad/website-status.git",
        "clone_url": "https://github.com/Real-Dev-Squad/website-status.git",
        "svn_url": "https://github.com/Real-Dev-Squad/website-status",
        "homepage": "https://status.realdevsquad.com/",
        "size": 5138,
        "stargazers_count": 15,
        "watchers_count": 15,
        "language": "TypeScript",
        "has_issues": true,
        "has_projects": true,
        "has_downloads": true,
        "has_wiki": true,
        "has_pages": false,
        "has_discussions": false,
        "forks_count": 108,
        "mirror_url": null,
        "archived": false,
        "disabled": false,
        "open_issues_count": 137,
        "license": {
          "key": "mit",
          "name": "MIT License",
          "spdx_id": "MIT",
          "url": "https://api.github.com/licenses/mit",
          "node_id": "MDc6TGljZW5zZTEz"
        },
        "allow_forking": true,
        "is_template": false,
        "web_commit_signoff_required": false,
        "topics": ["hacktoberfest", "react", "typescript"],
        "visibility": "public",
        "forks": 108,
        "open_issues": 137,
        "watchers": 15,
        "default_branch": "develop"
      }
    },
    "base": {
      // Same as above head object
    },
    "_links": {
      "self": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/576"
      },
      "html": {
        "href": "https://github.com/Real-Dev-Squad/website-status/pull/576"
      },
      "issue": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues/576"
      },
      "comments": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/issues/576/comments"
      },
      "review_comments": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/576/comments"
      },
      "review_comment": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/comments{/number}"
      },
      "commits": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/pulls/576/commits"
      },
      "statuses": {
        "href": "https://api.github.com/repos/Real-Dev-Squad/website-status/statuses/087cf71ba3b38237a3e5217a9ab3816a871f3849"
      }
    },
    "author_association": "CONTRIBUTOR",
    "auto_merge": null,
    "active_lock_reason": null
  }
]

Field which are required in the PR

[x] url - URL of the PR in the browser
[ ] id - ID of the PR it is unique for every PR
[ ] node_id - Node ID of the PR it is unique for every PR
[x] html_url - URL of the PR in the browser
[ ] diff_url - Used to get the diff of the PR
[ ] patch_url - Used to get the patch of the PR
[x] issue_url - URL of the issue in the browser
[x] number - Number of the PR it is unique for every PR
[x] state - State of the PR like open or closed
[ ] locked - If the PR is locked or not
[x] title - Title of the PR

PR user

[x] login - Username of the user
[x] id - ID of the user it is unique for every user
[ ] node_id - Node ID of the user it is unique for every user
[x] avatar_url - URL of the user's avatar
[ ] gravatar_id - Gravatar ID of the user
[x] url - URL of the user(API)
[x] html_url - URL of the user in the browser
[ ] followers_url - URL of the user's followers(API)
[ ] following_url - URL of the user's following(API)
[ ] gists_url - URL of the user's gists(API)
[ ] starred_url - URL of the user's starred gists(API)
[ ] subscriptions_url - URL of the user's subscriptions(API)
[ ] organizations_url - URL of the user's organizations(API)
[ ] repos_url - URL of the user's repos(API)
[ ] events_url - URL of the user's events(API)
[ ] received_events_url - URL of the user's received events(API)
[ ] type - Type of the user like User or Organization
[ ] site_admin - If the user is site admin or not

Continue to PR

[x] body - Body of the PR
[x] created_at - Date and time when the PR was created
[x] updated_at - Date and time when the PR was updated
[x] closed_at - Date and time when the PR was closed
[x] merged_at - Date and time when the PR was merged
[ ] merge_commit_sha - SHA of the merge commit

PR Assignee

Same as PR user

PR Assignees - Array

Same as PR Assignee but in array

Continue to PR

[ ] requested_reviewers - Array of the requested reviewers
[ ] requested_teams - Array of the requested teams
[x] labels - Array of the labels
[ ] milestone - Milestone of the PR
[x] draft - If the PR is draft or not
[ ] commits_url - URL of the commits(API)
[ ] review_comments_url - URL of the review comments(API)
[ ] review_comment_url - URL of the review comment(API)
[ ] comments_url - URL of the comments(API)
[ ] statuses_url - URL of the statuses(API)

PR Head

[x] label - Label of the head
[x] ref - Ref of the head
[ ] sha - SHA of the head
[ ] USER - User of the head

PR Head User

Same as PR user

PR Head Repo

[ ] id - ID of the head repo it is unique for every head repo
[ ] node_id - Node ID of the head repo it is unique for every head repo
[x] name - Name of the head repo
[x] full_name - Full name of the head repo
[x] private - If the head repo is private or not

PR Head Repo Owner

Same as PR user and PR head user

Continue to PR Head Repo

[x] html_url - URL of the head repo in the browser
[x] description - Description of the head repo
[ ] fork - If the head repo is fork or not
[ ] url - URL of the head repo(API)
[ ] forks_url - URL of the head repo's forks(API)
[ ] keys_url - URL of the head repo's keys(API)
[ ] collaborators_url - URL of the head repo's collaborators(API)
[ ] teams_url - URL of the head repo's teams(API)
[ ] hooks_url - URL of the head repo's hooks(API)
[ ] issue_events_url - URL of the head repo's issue events(API)
[ ] events_url - URL of the head repo's events(API)
[ ] assignees_url - URL of the head repo's assignees(API)
[ ] branches_url - URL of the head repo's branches(API)
[ ] tags_url - URL of the head repo's tags(API)
[ ] blobs_url - URL of the head repo's blobs(API)
[ ] git_tags_url - URL of the head repo's git tags(API)
[ ] git_refs_url - URL of the head repo's git refs(API)
[ ] trees_url - URL of the head repo's trees(API)
[ ] statuses_url - URL of the head repo's statuses(API)
[ ] languages_url - URL of the head repo's languages(API)
[ ] stargazers_url - URL of the head repo's stargazers(API)
[ ] contributors_url - URL of the head repo's contributors(API)
[ ] subscribers_url - URL of the head repo's subscribers(API)
[ ] subscription_url - URL of the head repo's subscription(API)
[ ] commits_url - URL of the head repo's commits(API)
[ ] git_commits_url - URL of the head repo's git commits(API)
[ ] comments_url - URL of the head repo's comments(API)
[ ] issue_comment_url - URL of the head repo's issue comment(API)
[ ] contents_url - URL of the head repo's contents(API)
[ ] compare_url - URL of the head repo's compare(API)
[ ] merges_url - URL of the head repo's merges(API)
[ ] archive_url - URL of the head repo's archive(API)
[ ] downloads_url - URL of the head repo's downloads(API)
[ ] issues_url - URL of the head repo's issues(API)
[ ] pulls_url - URL of the head repo's pulls(API)
[ ] milestones_url - URL of the head repo's milestones(API)
[ ] notifications_url - URL of the head repo's notifications(API)
[ ] labels_url - URL of the head repo's labels(API)
[ ] releases_url - URL of the head repo's releases(API)
[ ] deployments_url - URL of the head repo's deployments(API)
[ ] created_at - Created at of the head repo
[ ] updated_at - Updated at of the head repo
[ ] pushed_at - Pushed at of the head repo
[ ] git_url - Git URL of the head repo
[ ] ssh_url - SSH URL of the head repo
[ ] clone_url - Clone URL of the head repo
[ ] svn_url - SVN URL of the head repo
[ ] homepage - Homepage of the head repo
[ ] size - Size of the head repo
[ ] stargazers_count - Stargazers count of the head repo
[ ] watchers_count - Watchers count of the head repo
[ ] language - Language of the head repo
[ ] has_issues - If the head repo has issues or not
[ ] has_projects - If the head repo has projects or not
[ ] has_downloads - If the head repo has downloads or not
[ ] has_wiki - If the head repo has wiki or not
[ ] has_pages - If the head repo has pages or not
[ ] has_discussions - If the head repo has discussions or not
[ ] forks_count - Forks count of the head repo
[ ] mirror_url - Mirror URL of the head repo
[ ] archived - If the head repo is archived or not
[ ] disabled - If the head repo is disabled or not
[ ] open_issues_count - Open issues count of the head repo

PR Head Repo Owner License

[ ] key - Key of the head repo owner license
[ ] name - Name of the head repo owner license
[ ] spdx_id - SPDX ID of the head repo owner license
[ ] url - URL of the head repo owner license
[ ] node_id - Node ID of the head repo owner license

Continue to PR Head Repo

[ ] allow_forking - If the head repo is allowed to fork or not
[ ] is_template - If the head repo is template or not
[ ] topics - Topics of the head repo - Array
[ ] visibility - Visibility of the head repo
[ ] forks - Forks of the head repo
[ ] open_issues - Open issues of the head repo
[ ] watchers - Watchers of the head repo
[ ] default_branch - Default branch of the head repo

PR base

Same as PR head

PR _links

[ ] self - Self of the PR
[ ] html - HTML of the PR
[ ] issue - Issue of the PR
[ ] comments - Comments of the PR
[ ] review_comments - Review comments of the PR
[ ] review_comment - Review comment of the PR
[ ] commits - Commits of the PR
[ ] statuses - Statuses of the PR

Continue to PR

[ ] author_association - Author association of the PR
[ ] auto_merge - Auto merge of the PR
[ ] active_lock_reason - Active lock reason of the PR

RitikJaiswal75 commented 1 year ago

Please mention the reason why we want to move all the pr data to rds

remember once @harshith-venkatesh told us that we can set up rules for naming PRs and based on that we can filter it out.

Real-Dev-Squad / website-backend

[RFC]: Store Org's all Pull Requests in Real Dev Squad Backend Database and Webhook Integration #1133

[RFC]: Store Org's all Pull Requests in Real Dev Squad Backend Database and Webhook Integration

Summary

Motivation

Detailed design

1. Introduction

2. Background

3. Objectives

4. Proposed Solution

4.1. Database Schema

4.2. API Endpoint

4.3. Webhook Integration

4.4. Event Processing

4.5. Error Handling

5. Security Considerations

6. Future Enhancements

Drawbacks

Unresolved questions

Conclusion

Field which are required in the PR

PR user

Continue to PR

PR Assignee

PR Assignees - Array

Continue to PR

PR Head

PR Head User

PR Head Repo

PR Head Repo Owner

Continue to PR Head Repo

PR Head Repo Owner License

Continue to PR Head Repo

PR base

PR _links

Continue to PR