astral-sh / ruff

An extremely fast Python linter and code formatter, written in Rust.
https://docs.astral.sh/ruff
MIT License
32.5k stars 1.08k forks source link

Adding new class of static analysis rules using LLMs #14085

Open bbkgh opened 1 day ago

bbkgh commented 1 day ago

Hi, I would like to suggest adding a new class of rules to Ruff that utilizes Local/Remote LLM APIs for running rules over code files. I believe this can greatly improve code quality and result in fewer bugs. Additionally, writing rules in plain text is more convenient. For example, we could add a llm_rules.yaml file to project containing something like this:

type: ollama/qwen2.5
endpoint: http://localhost:11434
rules:
  - "There must not be any usage of datetime.now(); use timezone.now() from django.utils instead"
  - "There must not be any Typos in comments"
  - "Always use MyCustomObject for calling ServiceX"
MichaReiser commented 1 day ago

Using LLMs for analysis is certainly interesting but I'm not sure if asking the LLM to do the analysis is the solution because LLMs are much slower than regular rules. Instead, we should explore if it's possible for the LLM to derive the necessary checks once ahead of time that Ruff can then run as part of its engine. This also removes any need to directly integrate with an LLVM in ruff itself.

I also think LLMs should be used very sparsely. E.g. the first rule is already covered by Ruff.