Use "chain of thought" instead of single prompt to the language model

[h1alexbel]

We should use chain of thought prompting strategy instead the one we use now: just one prompt to the LLM in order to get the result

[github-actions[bot]]

@h1alexbel thanks for detailed and disciplined report.

[h1alexbel]

the context comliing:

[
  "The bug report lacks a clear summary or title.",
  "The report is concise, but it would be better if it provided more context or background information, such as the version of the software, when this issue was first discovered, or any specific use cases where this problem occurs.",
  "The issue is not explicitly labeled as a bug or issue, which could make it confusing for developers who might not be familiar with the project's terminology. Labeling it as a 'Grammar issue' or 'ANTLR typo' could provide more clarity.",
  "Mentioning that this is an issue in a grammar file could be more clear if it specified that it is an ANTLR grammar file, as not all grammar files use ANTLR. This information could help developers who are not familiar with the specific grammar used in the project.",
  "The report does not provide any information about the potential impact of this issue on the software's functionality or any suggestions for how to resolve it. Including details about how the incorrect grammar might cause the software to behave incorrectly or fail to parse certain inputs correctly would be helpful for developers trying to diagnose and fix the problem."
]

[h1alexbel]

input bug report:

`deltaBidning` and `lambdaBidning` Typos in Phi.g4 grammar file:
There are typos in `deltaBidning` and `lambdaBidning` in our Phi.g4 grammar.

take a look here
binding
    : alphaBinding
    | emptyBinding
    | deltaBidning
    | lambdaBidning
    ;
It should be deltaBinding and lambdaBinding instead

problems.json:

[
      "The bug report lacks a clear and concise description of the issue.",
      "The bug report does not provide any steps to reproduce the issue.",
      "The bug report does not specify the expected and actual behavior.",
      "The bug report does not mention the software version or environment where the issue occurred.",
      "The bug report does not have a proper title that summarizes the issue."
]

contexted.json:

[
    "The bug report lacks a clear and concise description of the issue. In this case, the issue is related to typos in the Phi.g4 grammar file.",
    "The bug report does not provide any steps to reproduce the issue. To reproduce the issue, the developers should search for the terms 'deltaBidning' and 'lambdaBidning' in the Phi.g4 grammar file and verify if the typos are present.",
    "The bug report does not specify the expected and actual behavior. The expected behavior is for the grammar file to use the correct terms 'deltaBinding' and 'lambdaBinding'. The actual behavior is the current presence of the typos 'deltaBidning' and 'lambdaBidning'.",
    "The bug report does not mention the software version or environment where the issue occurred. This information is crucial for the developers to reproduce and resolve the issue effectively.",
    "The bug report does not have a proper title that summarizes the issue. A better title for this bug report could be 'Typos in deltaBidning and lambdaBidning in Phi.g4 grammar file'."
]

top.json:

[
    "The bug report lacks a clear and concise description of the issue. In this case, the issue is related to typos in the Phi.g4 grammar file.",
    "The bug report does not provide any steps to reproduce the issue. To reproduce the issue, the developers should search for the terms 'deltaBidning' and 'lambdaBidning' in the Phi.g4 grammar file and verify if the typos are present.",
    "The bug report does not specify the expected and actual behavior. The expected behavior is for the grammar file to use the correct terms 'deltaBinding' and 'lambdaBinding'. The actual behavior is the current presence of the typos 'deltaBidning' and 'lambdaBidning'."
]

as you can see, we have false positives in problems.json:

The bug report does not specify the expected and actual behavior.

Then it was chained in contexted.json:

The bug report does not specify the expected and actual behavior. The expected behavior is for the grammar file to use the correct terms 'deltaBinding' and 'lambdaBinding'. The actual behavior is the current presence of the typos 'deltaBidning' and 'lambdaBidning'.

[h1alexbel]

This is the summary of the latest experiment with the "Chain of Thought" prompting. The strategy was to:

1. Identify the quality problems
2. Embed them with context
3. Pick the most important problems (max amount is 3)
4. Generate suggestions

The results so far are saying that there is a big chance of generating false positives in quality problems. To tackle this problem, we are trying to input rules along with the report on step no. 1. This solution also provokes us to do one more prompt: rule matching with the bug report.

[h1alexbel]

we also can do verifications after receiving all the problems

[h1alexbel]

chain logic was a bit refined after reading this paper. LLMs so far are struggling with hallucination and stochasticity. To tackle that, let's change our logic with prompting. Now we will work only with text and give additional prompts to the model to format the response to JSON array. That's why I created #70. Besides that we need to empower our strategy with a technique called self-validation.

Here is the refined pipeline:

report:

`deltaBidning` and `lambdaBidning` Typos in Phi.g4 grammar file:
There are typos in `deltaBidning` and `lambdaBidning` in our Phi.g4 grammar.

take a look here
binding
    : alphaBinding
    | emptyBinding
    | deltaBidning
    | lambdaBidning
    ;
It should be deltaBinding and lambdaBinding instead

problems:

Quality problems related to this bug report formulation:

* The title of the bug report "deltaBidning" and "lambdaBidning" Typos in Phi.g4 grammar file is not clear and descriptive enough. A better title could be "Incorrect naming in Phi.g4 grammar file".
* The bug report lacks a clear problem statement. It's suggested to start the description with a clear problem statement instead of an action. For example: "In the Phi.g4 grammar file, there are two typos in the following code block. Instead of 'deltaBidning' and 'lambdaBidning', it should be 'deltaBinding' and 'lambdaBinding'."
* The bug report lacks information about the impact of these typos on the software's functionality. For example, it would be helpful to know if the issue prevents compilation, causes runtime errors, or leads to unexpected behavior. Additionally, a severity label for the issue could be useful.

settings: temperature=0.7, max_new_tokens=512, system_prompt=quality-expert.ts

validated:

The title of the bug report "deltaBidning" and "lambdaBidning" Typos in Phi.g4 grammar file is not clear and descriptive enough. A better title could be "Incorrect naming in Phi.g4 grammar file".
The bug report lacks information about the impact of these typos on the software's functionality. For example, it would be helpful to know if the issue prevents compilation, causes runtime errors, or leads to unexpected behavior. Additionally, a severity label for the issue could be useful.

settings: temperature=1.0, max_new_tokens=1024, system_prompt=default

validated.json:

{
  "size": 2,
  "problems": [
    "The title of the bug report \"deltaBidning\" and \"lambdaBidning\" Typos in Phi.g4 grammar file is not clear and descriptive enough. A better title could be \"Incorrect naming in Phi.g4 grammar file\". The bug report lacks information about the impact of these typos on the software's functionality. For example, it would be helpful to know if the issue prevents compilation, causes runtime errors, or leads to unexpected behavior. Additionally, a severity label for the issue could be useful.",
    "The second problem statement goes here, if any."
  ]
}

settings: TBD..

top:

input:

{
  "size": 6,
  "problems": [
    "1. Lack of a clear description: The report lacks a clear and concise description of the problem. It simply states there are typos but does not specify what the typos are or how they impact the system.",
    "2. Missing steps to reproduce: There are no steps provided to reproduce the issue. This makes it difficult for developers to identify if they have fixed the issue correctly.",
    "3. No severity level: The severity level of the issue is not stated. This is important information for developers to prioritize how soon the issue should be resolved.",
    "4. Lack of environment details: The report does not mention which environment this issue occurs in (e.g., which version of the software, which operating system).",
    "5. Use of shorthand: The term 'take a look here' is used, which is not clear or professional. It is best to avoid using shorthand or colloquial language in formal documentation.",
    "6. Incomplete code block: The code block is not complete (it is cut off after the relevant lines). This makes it difficult for developers to understand the context of the issue."
  ]
}

result:

{
  "size": 3,
  "problems": [
    "1. Lack of a clear description: The report lacks a clear and concise description of the problem. It simply states there are typos but does not specify what the typos are or how they impact the system.",
    "2. Missing steps to reproduce: There are no steps provided to reproduce the issue. This makes it difficult for developers to identify if they have fixed the issue correctly.",
    "6. Incomplete code block: The code block is not complete (it is cut off after the relevant lines). This makes it difficult for developers to understand the context of the issue."
  ]
}

polished.json:

{
  "size": 3,
  "problems": [
    "Lack of a clear description: The report lacks a clear and concise description of the problem. It simply states there are typos but does not specify what the typos are or how they impact the system.",
    "Missing steps to reproduce: There are no steps provided to reproduce the issue. This makes it difficult for developers to identify if they have fixed the issue correctly.",
    "Incomplete code block: The code block is not complete (it is cut off after the relevant lines). This makes it difficult for developers to understand the context of the issue."
  ]
}

suggestions:

Suggested improvements:

Clear description: Please provide a clear explanation of the issue. Explain what typos were found in the code (deltaBidning and lambdaBidning) and how they impact the system functionality.

Steps to reproduce: Include the necessary steps to reproduce the issue. This will help the developers understand the issue better and identify if the fix was correct.

Complete code block: Kindly provide the complete code block for better context understanding.

Suggested Bug Report:

deltaBidning and lambdaBidning Typos in Phi.g4 grammar file: In our Phi.g4 grammar file, we have identified typos in the following lines:

binding
    : alphaBinding
    | emptyBinding
    | deltaBidning
    | lambdaBidning
    ;
The correct syntax should be:

binding
    : alphaBinding
    | emptyBinding
    | deltaBinding
    | lambdaBinding
    ;
The typos(deltaBidning and lambdaBidning) impact the system's syntax understanding and cause unexpected behavior or errors.

To reproduce this issue, follow these steps:

Open the Phi.g4 grammar file.
Locate the lines marked above.
Observe that deltaBidning should be deltaBinding and lambdaBidning should be lambdaBinding.
These corrections will solve the issue and appropriately define the grammar rules for binding.

suggestions.json:

{
  "suggestions": [
    "Clear description: Please provide a clear explanation of the issue. Explain what typos were found in the code (deltaBidning and lambdaBidning) and how they impact the system functionality.",
    "Steps to reproduce: Include the necessary steps to reproduce the issue. This will help the developers understand the issue better and identify if the fix was correct.",
    "Complete code block: Kindly provide the complete code block for better context understanding.",
    "Suggested Bug Report:deltaBidning and lambdaBidning Typos in Phi.g4 grammar file: In our Phi.g4 grammar file, we have identified typos in the following lines:\n\n```\nbinding\n    : alphaBinding\n    | emptyBinding\n    | deltaBidning\n    | lambdaBidning\n    ;\n```\nThe correct syntax should be:\n\n```\nbinding\n    : alphaBinding\n    | emptyBinding\n    | deltaBinding\n    | lambdaBinding\n    ;\n```\nThe typos(deltaBidning and lambdaBidning) impact the system's syntax understanding and cause unexpected behavior or errors.\n\nTo reproduce this issue, follow these steps:\n\n- Open the Phi.g4 grammar file.\n- Locate the lines marked above.\n- Observe that deltaBidning should be deltaBinding and lambdaBidning should be lambdaBinding.\n\nThese corrections will solve the issue and appropriately define the grammar rules for binding."
  ]
}