160 added test framework and proper error messaging in prompt

EmbeddedDevops1 commented 2 months ago

PR Type

enhancement, tests

Description

Added a testing_framework parameter to the PromptBuilder class and integrated it into prompt generation.
Enhanced error handling in UnitTestGenerator by improving error message extraction and updating error messages for failed tests.
Introduced a new test case test_failing_test to the test suite.
Updated the test command and configuration settings in increase_coverage.py.
Added a new template for analyzing test run failures in analyze_test_run_failure.toml.
Updated the test generation prompt to include test framework details.
Bumped the version from 0.1.50 to 0.1.51.

Changes walkthrough 📝

Relevant files

Enhancement

PromptBuilder.py `Add testing framework parameter to prompt builder` cover_agent/PromptBuilder.py Added `testing_framework` parameter to the constructor. Included `testing_framework` in the prompt building methods.	+5/-0
UnitTestGenerator.py `Enhance error handling and testing framework integration` cover_agent/UnitTestGenerator.py Initialized `testing_framework` attribute. Improved error message extraction. Updated error messages for failed tests.	+8/-6
analyze_test_run_failure.toml `Add template for test run failure analysis` cover_agent/settings/analyze_test_run_failure.toml - Added a new template for analyzing test run failures.	+41/-0
test_generation_prompt.toml `Update test generation prompt with framework details` cover_agent/settings/test_generation_prompt.toml - Included test framework information in the prompt.	+2/-0

Tests

test_UnitTestDB.py

Add failing test case for testing purposes

tests/test_UnitTestDB.py - Added a new test `test_failing_test`.

+3/-0

Configuration changes

increase_coverage.py

Update test command and configuration settings

tests_integration/increase_coverage.py

Modified test command and reduced max iterations.

Removed additional instructions.

+3/-3

Miscellaneous

version.txt

Bump version to 0.1.51

cover_agent/version.txt - Updated version from 0.1.50 to 0.1.51.

+1/-1

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

codiumai-pr-agent-pro[bot] commented 2 months ago

CI Failure Feedback 🧐

(Checks updated until commit https://github.com/Codium-ai/cover-agent/commit/c102ca33647e9c4ec937fe363d36391a75f49151)

**Action:** run-integration-tests

**Failed stage:** [Run integration tests](https://github.com/Codium-ai/cover-agent/actions/runs/11046084198/job/30684883077) [❌]

**Failure summary:** The action failed due to a FileNotFoundError in the script execution:

The script attempted to access a settings file named analyze_test_run_failure.toml.

The file was expected to be located at /tmp/_MEIG2fhIS/analyze_test_run_failure.toml.

The absence of this file caused the script to terminate with an unhandled exception.

Relevant error logs:

```yaml 1: ##[group]Operating System 2: Ubuntu ... 191: ##[group]Setting up auth 192: [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 193: [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 194: [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 195: [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 196: [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 197: ##[endgroup] 198: ##[group]Fetching the repository 199: [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +c102ca33647e9c4ec937fe363d36391a75f49151:refs/remotes/origin/160-added-test-framework-and-proper-error-messaging-in-prompt ... 509: Resolving deltas: 16% (1/6) 510: Resolving deltas: 33% (2/6) 511: Resolving deltas: 50% (3/6) 512: Resolving deltas: 66% (4/6) 513: Resolving deltas: 83% (5/6) 514: Resolving deltas: 100% (6/6) 515: Resolving deltas: 100% (6/6), done. 516: From https://github.com/Codium-ai/cover-agent 517: * [new ref] c102ca33647e9c4ec937fe363d36391a75f49151 -> origin/160-added-test-framework-and-proper-error-messaging-in-prompt 518: ##[endgroup] 519: ##[group]Determining the checkout info 520: ##[endgroup] 521: ##[group]Checking out the ref 522: [command]/usr/bin/git checkout --progress --force -B 160-added-test-framework-and-proper-error-messaging-in-prompt refs/remotes/origin/160-added-test-framework-and-proper-error-messaging-in-prompt 523: branch '160-added-test-framework-and-proper-error-messaging-in-prompt' set up to track 'origin/160-added-test-framework-and-proper-error-messaging-in-prompt'. 524: Switched to a new branch '160-added-test-framework-and-proper-error-messaging-in-prompt' ... 717: Traceback (most recent call last): 718: File "cover_agent/main.py", line 114, in 719: File "cover_agent/main.py", line 109, in main 720: File "cover_agent/CoverAgent.py", line 29, in __init__ 721: File "cover_agent/UnitTestGenerator.py", line 66, in __init__ 722: File "cover_agent/UnitTestGenerator.py", line 110, in get_code_language 723: File "cover_agent/settings/config_loader.py", line 57, in get_settings 724: File "cover_agent/settings/config_loader.py", line 49, in __init__ 725: FileNotFoundError: Settings file not found: /tmp/_MEIG2fhIS/analyze_test_run_failure.toml 726: [PYI-13:ERROR] Failed to execute script 'main' due to unhandled exception! 727: + cleanup 728: Cleaning up... 729: + '[' -n cf9b48e1ea6d7bfab575070f359e674c043e114333fa9921d891ec8175463479 ] 730: + echo 'Cleaning up...' 731: + docker stop cf9b48e1ea6d7bfab575070f359e674c043e114333fa9921d891ec8175463479 732: cf9b48e1ea6d7bfab575070f359e674c043e114333fa9921d891ec8175463479 733: + docker rm cf9b48e1ea6d7bfab575070f359e674c043e114333fa9921d891ec8175463479 734: cf9b48e1ea6d7bfab575070f359e674c043e114333fa9921d891ec8175463479 735: ##[error]Process completed with exit code 1. ```

✨ CI feedback usage guide:

The CI feedback tool (`/checks)` automatically triggers when a PR has a failed check. The tool analyzes the failed checks and provides several feedbacks: - Failed stage - Failed test name - Failure summary - Relevant error logs In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR: ``` /checks "https://github.com/{repo_name}/actions/runs/{run_number}/job/{job_number}" ``` where `{repo_name}` is the name of the repository, `{run_number}` is the run number of the failed check, and `{job_number}` is the job number of the failed check. #### Configuration options - `enable_auto_checks_feedback` - if set to true, the tool will automatically provide feedback when a check is failed. Default is true. - `excluded_checks_list` - a list of checks to exclude from the feedback, for example: ["check1", "check2"]. Default is an empty list. - `enable_help_text` - if set to true, the tool will provide a help message with the feedback. Default is true. - `persistent_comment` - if set to true, the tool will overwrite a previous checks comment with the new feedback. Default is true. - `final_update_message` - if `persistent_comment` is true and updating a previous checks message, the tool will also create a new message: "Persistent checks updated to latest commit". Default is true. See more information about the `checks` tool in the [docs](https://pr-agent-docs.codium.ai/tools/ci_feedback/).

codiumai-pr-agent-pro[bot] commented 2 months ago

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Key issues to review

Failing Test
A new test case 'test_failing_test' was added that intentionally fails. This might affect the overall test suite and should be reviewed. Configuration Change
The test command was modified and the maximum iterations were reduced from 5 to 2. This could potentially impact the coverage and effectiveness of the integration tests.

codiumai-pr-agent-pro[bot] commented 2 months ago

PR Code Suggestions ✨

Category	Suggestion	Score
Best practice	✅ Replace intentionally failing test with a meaningful test case ___ Suggestion Impact: The intentionally failing test was removed, which aligns with the suggestion to replace it with a meaningful test case. code diff: ```diff - def test_failing_test(self): - assert False ``` ___ The `test_failing_test` method is intentionally failing, which is not a good practice for unit tests. Consider removing this test or replacing it with a meaningful test case that checks for expected behavior. [tests/test_UnitTestDB.py [95-96]](https://github.com/Codium-ai/cover-agent/pull/165/files#diff-ef906a6dc974d145fc1da0019a94c057af2bb7319ee7efa2f93b1ba3fc0fcaf2R95-R96) ```diff -def test_failing_test(self): - assert False +def test_example_functionality(self): + # Replace with a meaningful test case + result = some_function() + assert result == expected_value ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 9 Why: Replacing an intentionally failing test with a meaningful test case is crucial for ensuring that unit tests provide value by checking expected behavior, thus improving test quality.	9
Maintainability	✅ Extract error message generation into a separate method for better code organization ___ Suggestion Impact: The error message generation logic was extracted into a separate method named extract_error_message, improving code organization. code diff: ```diff + error_message = self.extract_error_message(stderr=fail_details["stderr"], stdout=fail_details["stdout"]) if error_message: - logging.error(f"Error message:\n{error_message}") + logging.error(f"Error message summary:\n{error_message}") self.failed_test_runs.append( {"code": generated_test, "error_message": error_message} @@ -764,30 +762,43 @@ return json.dumps(self.to_dict()) -def extract_error_message_python(fail_message): - """ - Extracts and returns the error message from the provided failure message. - - Parameters: - fail_message (str): The failure message containing the error message to be extracted. - - Returns: - str: The extracted error message from the failure message, or an empty string if no error message is found. - - """ - try: - # Define a regular expression pattern to match the error message - MAX_LINES = 20 - pattern = r"={3,} FAILURES ={3,}(.?)(={3,}\|$)" - match = re.search(pattern, fail_message, re.DOTALL) - if match: - err_str = match.group(1).strip("\n") - err_str_lines = err_str.split("\n") - if len(err_str_lines) > MAX_LINES: - # show last MAX_lines lines - err_str = "...\n" + "\n".join(err_str_lines[-MAX_LINES:]) - return err_str - return "" - except Exception as e: - logging.error(f"Error extracting error message: {e}") - return ""+ def extract_error_message(self, stderr, stdout): + """ + Extracts the error message from the provided stderr and stdout outputs. + + Updates the PromptBuilder object with the stderr and stdout, builds a custom prompt for analyzing test run failures, + calls the language model to analyze the prompt, and loads the response into a dictionary. + + Returns the error summary from the loaded YAML data or a default error message if unable to summarize. + Logs errors encountered during the process. + + Parameters: + stderr (str): The standard error output from the test run. + stdout (str): The standard output from the test run. + + Returns: + str: The error summary extracted from the response or a default error message if extraction fails. + """ + try: + # Update the PromptBuilder object with stderr and stdout + self.prompt_builder.stderr_from_run = stderr + self.prompt_builder.stdout_from_run = stdout + + # Build the prompt + prompt_headers_indentation = self.prompt_builder.build_prompt_custom( + file="analyze_test_run_failure" + ) + + # Run the analysis via LLM + response, prompt_token_count, response_token_count = ( + self.ai_caller.call_model(prompt=prompt_headers_indentation, stream=False + ) + self.total_input_token_count += prompt_token_count + self.total_output_token_count += response_token_count + tests_dict = load_yaml(response) + + return tests_dict.get("error_summary", f"ERROR: Unable to summarize error message from inputs. STDERR: {stderr}\nSTDOUT: {stdout}.") + except Exception as e: + logging.error(f"ERROR: Unable to extract error message from inputs using LLM.\nSTDERR: {stderr}\nSTDOUT: {stdout}\n\n{response}") + logging.error(f"Error extracting error message: {e}") + return "" ``` ___ Consider extracting the error message generation logic into a separate method for better code organization and potential reuse.* [cover_agent/UnitTestGenerator.py [565-570]](https://github.com/Codium-ai/cover-agent/pull/165/files#diff-19760582d9ede3a799fdbb541ad357b4822682e837bca8365196fba50daf57e3R565-R570) ```diff # TODO: Get LLM to summarize the error by passing it the stdout and stderr # Use `cover_agent/settings/analyze_test_run_failure.toml` template -error_message = extract_error_message_python(fail_details["stdout"]) +error_message = self.generate_error_summary(fail_details["stdout"], fail_details["stderr"]) if error_message: logging.error(f"Error message:\n{error_message}") +def generate_error_summary(self, stdout, stderr): + # Implement error summary generation logic here + # Use the analyze_test_run_failure.toml template + pass + ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 8 Why: Extracting the error message generation logic into a separate method enhances code organization and reusability, which is a significant improvement for maintainability.	8
Enhancement	Improve variable naming for better code readability and understanding ___ Consider using a more descriptive variable name instead of `tests_dict`. For example, `test_analysis_result` would better convey the purpose and content of this dictionary. [cover_agent/UnitTestGenerator.py [359-365]](https://github.com/Codium-ai/cover-agent/pull/165/files#diff-19760582d9ede3a799fdbb541ad357b4822682e837bca8365196fba50daf57e3R359-R365) ```diff -tests_dict = self.analyze_test_suite(self.test_file_path) -relevant_line_number_to_insert_tests_after = tests_dict.get( +test_analysis_result = self.analyze_test_suite(self.test_file_path) +relevant_line_number_to_insert_tests_after = test_analysis_result.get( "relevant_line_number_to_insert_tests_after", None ) -relevant_line_number_to_insert_imports_after = tests_dict.get( +relevant_line_number_to_insert_imports_after = test_analysis_result.get( "relevant_line_number_to_insert_imports_after", None ) -self.testing_framework = tests_dict.get("testing_framework", "Unknown") +self.testing_framework = test_analysis_result.get("testing_framework", "Unknown") ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 7 Why: The suggestion to rename `tests_dict` to `test_analysis_result` improves code readability and understanding by providing a more descriptive name, which is beneficial for maintainability.	7

💡 Need additional feedback ? start a PR chat

EmbeddedDevops1 commented 2 months ago

⌛ Running regression testing: https://github.com/Codium-ai/cover-agent/actions/runs/11046084198

coditamar commented 2 months ago

/analyze

codiumai-pr-agent-pro[bot] commented 2 months ago

PR Analysis 🔬

This screen contains a list of code components that were changed in this PR.
You can initiate specific actions for each component, by checking the relevant boxes.
After you check a box, the action will be performed automatically by PR-Agent.
Results will appear as a comment on the PR, typically after 30-60 seconds.

file

Changed components

AICaller.py

- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar

call_model

^{(method of AICaller)}

+6/-6

PromptBuilder.py

- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar	__init__ ^{(method of PromptBuilder)}	+6/-0
- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar	build_prompt ^{(method of PromptBuilder)}	+4/-1
- [ ] Test - [ ] Docs - [ ] Improve - [x] Similar	build_prompt_custom ^{(method of PromptBuilder)}	+21/-7

UnitTestGenerator.py

- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar	__init__ ^{(method of UnitTestGenerator)}	+2/-1
- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar	build_prompt ^{(method of UnitTestGenerator)}	+3/-5
- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar	initial_test_suite_analysis ^{(method of UnitTestGenerator)}	+2/-1
- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar	validate_test ^{(method of UnitTestGenerator)}	+5/-5
- [ ] Test - [ ] Docs - [ ] Improve - [x] Similar	extract_error_message ^{(method of UnitTestGenerator)}	+40/-0

test_PromptBuilder.py

- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar

TestPromptBuilderEndToEnd

^(class)

+30/-0

increase_coverage.py

- [ ] Test - [ ] Docs - [ ] Improve - [ ] Similar

__init__

^(function)

+4/-4

💡 Usage guide:

Using static code analysis capabilities, the `analyze` tool scans the PR code changes and find the code components (methods, functions, classes) that changed in the PR. The tool can be triggered [automatically](https://pr-agent-docs.codium.ai/usage-guide/automations_and_usage/#github-app-automatic-tools-when-a-new-pr-is-opened) every time a new PR is opened, or can be invoked manually by commenting on any PR: ``` /analyze ``` Language that are currently supported: Python, Java, C++, JavaScript, TypeScript, C#. See more information about the tool in the [docs](https://pr-agent-docs.codium.ai/tools/analyze/).

codiumai-pr-agent-pro[bot] commented 2 months ago

🔍 Finding similar code for 'extract_error_message'

extract_error_message (method) ^[+40/-0]

Component signature:

def extract_error_message(self, stderr, stdout):

🏷 Extracted Keywords	error , summary , analyze , prompt , response
🌍 Search Context	- [x] Global - [ ] Organization
🔍 Similar Code	Repository: turbo-code/llmware Filename: prompts.py
	Repository: JoshuaOliphant/chickadee Filename: chickadee.py
	Repository: 345ishaan/codemate Filename: agent.py
	Repository: nfsec/eshell Filename: eshell
	Repository: Akshathapatil1998/Assignment-2 Filename: main.py
📂 Relevant Repositories	No relevant repositories found

codiumai-pr-agent-pro[bot] commented 2 months ago

🔍 Finding similar code for 'build_prompt_custom'

build_prompt_custom (method) ^[+21/-7]

Component signature:

def build_prompt_custom(self, file) -> dict:

🏷 Extracted Keywords	prompt , render , template , settings , variables
🌍 Search Context	- [x] Global - [ ] Organization
🔍 Similar Code	Repository: AllenInstitute/render-python Filename: render.py
	Repository: CreativeDesigner3D/PyClone Filename: pyclone_props.py
	Repository: beancount/beanquery Filename: shell.py
	Repository: timothycrosley/thedom Filename: base.py
	Repository: MTG/freesound Filename: settings.py
📂 Relevant Repositories	Repository: jinja2 Description: A popular templating engine for Python, similar to the environment rendering used in `build_prompt_custom`.
	Repository: cookiecutter Description: A command-line utility that creates projects from templates, similar to how `build_prompt_custom` uses templates to generate prompts.
	Repository: prompt_toolkit Description: A library for building interactive command line applications, which may involve generating prompts similar to `build_prompt_custom`.

EmbeddedDevops1 commented 2 months ago

⌛ Re-running regression testing: https://github.com/Codium-ai/cover-agent/actions/runs/11055623841

Codium-ai / cover-agent