search

mattzcarey / code-review-gpt

Code review powered by LLMs (OpenAI GPT4, Sonnet 3.5) & Embeddings ⚡️ Improve code quality and catch bugs before you break production 🚀 Lives in your Github/GitLab/Azure DevOps CI
https://oriontools.ai
MIT License
1.55k stars 153 forks source link

[BUG] code review found 17 items only commented 3 on PR #328

Open joelsprunger opened 6 months ago

joelsprunger commented 6 months ago

Current Behavior:

I added LangSmith tracing to my github action.

    env:
      LANGCHAIN_TRACING_V2: "true"
      LANGCHAIN_ENDPOINT: "https://api.smith.langchain.com"
      LANGCHAIN_API_KEY: ${{ secrets.LANGCHAIN_API_KEY }}
      LANGCHAIN_PROJECT: "code_review_gpt"

In looking over the traces outputs for my large PR, I see 17 comments in 5 or 6 groups. However the Summary chain to generate emojis) only has 3 of these comments as input. These are the ones that go to the PR comments. The other 15 seem to be discarded.

Expected Behavior:

The PR comments should contain ALL the Risk items found in code review. Perhaps the emoji summarizer needs to randomly select the comments to add, or add a finite number of comments prioritized by risk level.

Steps To Reproduce:

Hmm. Tough. Big PR. Check the tracing.

Suggestions on the root cause

I am guessing the root cause is the finite context window for the summary. You can still limit the comments used in the summary, but add ALL comments to the PR.

joelsprunger commented 6 months ago

I suggest generating a github artifact code-review-gpt-details.md with all the PR comments.

If there are more than 3, the action could generate the artifact, and add a comment:

More comments are available in your github artifacts

Please follow these steps to view the artifact:

  1. Navigate to the Actions tab of this repository.
  2. Click on the workflow run that corresponds to your PR.
  3. Find the Artifacts section at the bottom of the workflow run page.
  4. Click on the code-review-gpt-details.md artifact
mattzcarey commented 6 months ago

Sounds like a great addition. When I built this I wanted to reduce the cognitive load on a human reviewer hence the whole ranking and limiting to 3 comments. Happy to generate an artifacts file that makes a lot of sense.