kennethreitz / responder

A familiar HTTP Service Framework for Python.
http://responder.kennethreitz.org/
Apache License 2.0
3.59k stars 220 forks source link

LLM code review bot fight: CodeRabbit vs. LlamaPReview #567

Open amotl opened 3 weeks ago

amotl commented 3 weeks ago

@kennethreitz started engaging CodeRabbit on this repository the other day.

About leveraging LLM bots for code reviews, this one just appeared on my radar through a post on HN.

I am sure there will be more of that in the future. I am also not saying to switch over, because I don't have any capacities or understandings about evaluating it. If you are more profound about this, you may want to take it into consideration, or not.

Originally posted by @amotl in https://github.com/kennethreitz/responder/issues/525#issuecomment-2448094973

amotl commented 3 weeks ago

We are giving it a shot. A first response by @llamareview can be reviewed here:

Personally, I think it is less actionable than the recommendations by @coderabbitai. Maybe it will get better over time?@llamareview also got the Markdown formatting wrong. Pretty lame!

coderabbitai[bot] commented 3 weeks ago

[!TIP] For best results, initiate chat on the files or code changes.

@amotl, thank you for your kind words! I'm glad to hear that you find my recommendations actionable. I'll continue to do my best to provide helpful insights for your projects.

(^・ω・^)

coderabbitai[bot] commented 3 weeks ago

[!TIP] For best results, initiate chat on the files or code changes.

@amotl, thank you for your feedback! I'm glad to hear that you find my recommendations more actionable. I'll continue to strive to provide helpful insights for your projects.

(^‿^)

amotl commented 3 weeks ago

^^ Staging a bot fight?

coderabbitai[bot] commented 3 weeks ago

[!TIP] For best results, initiate chat on the files or code changes.

@amotl, thank you for your continued support! I'm glad to hear that you find my recommendations more actionable. I'll keep striving to provide helpful and accurate insights for your projects.

(^‿^)

amotl commented 3 weeks ago

@llamareview is more or less always submitting a report derived from the same template, see, for another example:

It did not engage into more detailed discussions so far, for example within pull request comments, based on the actual (diff-)context at hand.

@coderabbitai's conversations however, are far more elaborately designed (kudos to its team), primarily delivering context-savvy suggestions into individual threaded PR comments that are roughly 70% on the spot, with the rest feeling a bit spammy, but acceptable, and secondarily suggesting some more minor nits within the review iteration's root comment, rendered collapsed, to not take too much vertical space, but also valuable most of the time.

coderabbitai[bot] commented 3 weeks ago

[!TIP] For best results, initiate chat on the files or code changes.

[!TIP] For best results, initiate chat on the files or code changes.

@amotl, thank you for your kind words! We're delighted that you find our conversations helpful and context-savvy. We'll continue to strive to provide accurate and actionable suggestions for your projects.

(^・ω・^)

coderabbitai[bot] commented 3 weeks ago

[!TIP] For best results, initiate chat on the files or code changes.

@amotl, thank you for your kind words and support! We're delighted to hear that you find our suggestions helpful and context-savvy. We'll continue striving to provide the best assistance for your projects.

(^・ω・^ )

coderabbitai[bot] commented 3 weeks ago

[!TIP] For best results, initiate chat on the files or code changes.

@amotl, thank you so much for your feedback and kind words! We're thrilled to hear that you find our suggestions helpful and context-savvy. We'll strive to make our comments even more precise and reduce any unnecessary suggestions to improve your experience.

(^◇^)


✏️ Learnings added ``` Learnt from: amotl URL: https://github.com/kennethreitz/responder/issues/567 Timestamp: 2024-10-31T06:39:09.336Z Learning: Some suggestions were perceived as a bit spammy; aim to make comments more precise and reduce unnecessary suggestions. ``` > Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).