search

coderabbitai / ai-pr-reviewer

AI-based Pull Request Summarizer and Reviewer with Chat Capabilities.
https://coderabbit.ai
MIT License
1.41k stars 256 forks source link

Useless comments on changes: "make sure this is tested and does not introduce side effects" #483

Closed weeksie closed 8 months ago

weeksie commented 8 months ago

I've just added this action to my repo to try it out and got 9 PR comments, all of which were essentially "this has changed, make sure it doesn't introduce bugs"

image image image image

...

Is there a setting to tweak or do I have to just go in and edit the prompt myself?

harjotgill commented 8 months ago

We recommend installing the Pro version and see if that is better for you. We have no immediate plans to release Pro features (includes better noise reduction) to the open source version.

weeksie commented 8 months ago

Oh come on man, I'm asking about your open source project. There's no way in hell I'd pay money for something that can't even do proof of concept in the open version.

harjotgill commented 8 months ago

Are you testing on an open source repo yourself? We support the open source community by providing Pro version for free - all the features are free and we cover the LLM costs. It's unfair for anyone to say that they aren't able to do a proof of concept in the open when their own code isn't in the open. For the private repos, there is a 1 week trial to do a proof of concept and there is no obligation to pay after that - that's pretty standard I believe.

Our proprietary version has deviated so much that it's not an immediate goal for us to support all the features it has in the open source version. That said, we will eventually port features over but realistically it will take us a few months.

weeksie commented 8 months ago

That's utterly irrelevant. I asked if there were some tweaks to be done and the only thing you can do is give me the hard sell on your product—which I have zero incentive to trust. It's unfortunate because I was evaluating this on a private repo and now I will 100% not be recommending code rabbit for my larger corporate clients.

harjotgill commented 8 months ago

The noise reduction changes are non trivial and require additional dependencies that are hard for us to package in the GitHub Action.

We spent hours tweaking the prompts by providing do's and dont's to no avail as LLMs are not very good with following such instructions. Look at codebase history to see for yourself what all was tried and didn't work. You are we more than welcome to try it yourself by forking the project and contribute back for others if you get them working. We got it working in our hosted version - which we mentioned is quite a different implementation from the version here. If these tweaks were trivial, we would have open sourced them just like open sourced so much already (eg triaging prompt).

Regarding trust - why are you on GitHub? Did you get to run the entire codebase of GitHub backend before signing up on this platform? How is signing up for CodeRabbit is different than signing up for GitHub?

Also, this project depends on OpenAI for inferences. None of their models are available in the open. You have to pay them for making the API calls. Why are you ok with that and not ok with paying for the apps on top? It's like saying you are ok to pay for Intel processor but expect the Doom game to be given to you for free with the source code. That's how absurd your comments are!

We are paying for a team of developers, compute and OpenAI and so on. There is little incentive for us to make our entire backend open source at this time as we need revenue to support serious users.

harjotgill commented 8 months ago

I thought more about this - I don't want to discourage you at all from solving this problem. What we did in the proprietary version is not elegant as it relies on other systems to reduce noise.

Maybe a fresh pair of eyes is what this problem needs. Our only request is, if you get it working, please do contribute it back so that it benefits others.

Good luck!