Detailed logging using databases

EmbeddedDevops1 commented 2 months ago

PR Type

enhancement, tests, documentation

Description

Integrated UnitTestDB for logging unit test generation attempts, enhancing the logging capabilities of the Cover Agent.
Added new command-line argument --log-db-path to specify the path for the log database.
Implemented tests for the UnitTestDB to ensure correct database operations.
Updated documentation to include instructions for using a test database and added a sequence diagram for better understanding of the system's workflow.
Added sqlalchemy as a dependency to manage database interactions.

Changes walkthrough 📝

Relevant files

Enhancement

CoverAgent.py `Integrate UnitTestDB for logging test attempts` cover_agent/CoverAgent.py Added `UnitTestDB` integration for logging test attempts. Created a default database file if not provided. Inserted test results into the database.	+13/-1
UnitTestDB.py `Implement UnitTestDB for managing test logs` cover_agent/UnitTestDB.py Implemented `UnitTestDB` class for database operations. Defined `UnitTestGenerationAttempt` model for test attempts. Added methods for inserting and querying test attempts.	+79/-0
UnitTestGenerator.py `Enhance UnitTestGenerator with serialization methods` cover_agent/UnitTestGenerator.py Added `llm_model` as a class variable. Implemented `to_dict` and `to_json` methods for serialization.	+19/-1
main.py `Add command-line argument for database path` cover_agent/main.py - Added `--log-db-path` argument for specifying database path.	+5/-0

Tests

test_UnitTestDB.py

Add tests for UnitTestDB functionality

tests/test_UnitTestDB.py

Added tests for UnitTestDB methods.

Verified database operations for test attempts.

+129/-0

Documentation

database_usage.md `Document database usage for test logging` docs/database_usage.md Documented usage of test database with Cover Agent. Provided instructions for running with an external DB.	+66/-0
top_level_sequence_diagram.md `Add sequence diagram for Cover Agent workflow` docs/top_level_sequence_diagram.md Added sequence diagram for Cover Agent's workflow. Illustrated interaction with LLM and other components.	+83/-1

Dependencies

pyproject.toml

Add SQLAlchemy dependency for database operations

pyproject.toml - Added `sqlalchemy` as a dependency.

+1/-0

💡 PR-Agent usage: Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

codiumai-pr-agent-pro[bot] commented 2 months ago

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 Security concerns

SQL injection:
The database connection string in UnitTestDB.py is directly used without sanitization (line 23). If this string includes user input, it could lead to SQL injection attacks. It's recommended to use parameterized queries or an ORM to mitigate this risk.

⚡ Key issues to review

Possible Security Issue
The database connection string is directly used without sanitization, which could lead to SQL injection if user input is involved. Error Handling
The code raises a FileNotFoundError if the DB file is not found, but it's not clear how this error is handled or if it's the appropriate way to handle a missing database file. Code Duplication
The select_attempt and select_attempt_flat methods have similar functionality and could potentially be combined to reduce code duplication.

codiumai-pr-agent-pro[bot] commented 2 months ago

PR Code Suggestions ✨

Category	Suggestion	Score
Best practice	Use a context manager for database connection to ensure proper resource management ___ Consider using a context manager for the database connection to ensure proper resource management and automatic closing of the connection. [cover_agent/CoverAgent.py [20]](https://github.com/Codium-ai/cover-agent/pull/151/files#diff-dac868643df79ec6dfcf446359468a88ba6a6617bb8ffa0139757f0fbf5195b1R20-R20) ```diff -self.test_db = UnitTestDB(db_connection_string=f"sqlite:///{args.log_db_path}") +with UnitTestDB(db_connection_string=f"sqlite:///{args.log_db_path}") as self.test_db: ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 7 Why:	7
Error handling	Use specific exception types for better error handling and diagnostics ___ Consider using a more specific exception type instead of a general `Exception` in the error handling for better error diagnostics and handling. [cover_agent/UnitTestDB.py [757-759]](https://github.com/Codium-ai/cover-agent/pull/151/files#diff-21ce07e5af653e42a56e74032ce3f94a6cc9d1f7d9223767fca4c04cec7472beR757-R759) ```diff -except Exception as e: +except (IndexError, ValueError) as e: logging.error(f"Error extracting error message: {e}") return "" ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 7 Why:	7
Testing best practice	Use a temporary database for testing to ensure test isolation ___ Consider using a temporary database for testing instead of a fixed file path to avoid potential conflicts and ensure test isolation. [tests/test_UnitTestDB.py [5]](https://github.com/Codium-ai/cover-agent/pull/151/files#diff-ef906a6dc974d145fc1da0019a94c057af2bb7319ee7efa2f93b1ba3fc0fcaf2R5-R5) ```diff -DATABASE_URL = 'sqlite:///unit_test_runs.db' +import tempfile +DATABASE_URL = f'sqlite:///{tempfile.mkstemp()[1]}' + ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 7 Why:	7
Security	Use double quotes for variable expansions to prevent word splitting and globbing ___ Consider using double quotes for variable expansions to prevent word splitting and globbing, especially when dealing with file paths that might contain spaces. [tests_integration/test_all.sh [46-48]](https://github.com/Codium-ai/cover-agent/pull/151/files#diff-65b6da080079e8f0585416d7e71797238296436ccac70a59aa5695b2a11a9d15R46-R48) ```diff if [ -n "$LOG_DB_PATH" ]; then - log_db_arg="--log-db-path $LOG_DB_PATH" + log_db_arg="--log-db-path \"$LOG_DB_PATH\"" fi ``` - [ ] Apply this suggestion Suggestion importance[1-10]: 7 Why:	7

codiumai-pr-agent-pro[bot] commented 2 months ago

CI Failure Feedback 🧐

(Checks updated until commit https://github.com/Codium-ai/cover-agent/commit/07c927f618ffa25721b3db4df21201060722e190)

**Action:** run-integration-tests

**Failed stage:** [Run integration tests](https://github.com/Codium-ai/cover-agent/actions/runs/10724314300/job/29739804818) [❌]

**Failed test name:** test_sqrt_negative_number_error

**Failure summary:** The action failed due to a NameError in the Python code. Specifically:

The function intended to return the square root of a number raised an error because the math module
was not imported.

The error occurred at line 2684 in app.py, where the code attempted to use math.sqrt(number) without
having imported the math module.

Relevant error logs:

```yaml 1: ##[group]Operating System 2: Ubuntu ... 823: double result = calculate(5, 3, "--modulus", &status); 824: TEST_ASSERT_EQUAL_INT(2, result); 825: TEST_ASSERT_EQUAL_INT(0, status); 826: } 827: new_imports_code: | 828: "" 829: test_tags: happy path 830: - test_behavior: | 831: Test that the function returns an error for modulus operation with zero as the second operand ... 851: double result = calculate(2, 3, "--power", &status); 852: TEST_ASSERT_EQUAL_DOUBLE(8.0, result); 853: TEST_ASSERT_EQUAL_INT(0, status); 854: } 855: new_imports_code: | 856: #include 857: test_tags: happy path 858: - test_behavior: | 859: Test that the function returns an error for an unknown operation ... 869: new_imports_code: | 870: "" 871: test_tags: edge case 872: 2024-09-05 16:27:33,102 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "sh build_and_test_with_coverage.sh" 873: 2024-09-05 16:27:33,778 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 50.0% 874: 2024-09-05 16:27:33,781 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "sh build_and_test_with_coverage.sh" 875: 2024-09-05 16:27:34,460 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 56.25% 876: 2024-09-05 16:27:34,464 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "sh build_and_test_with_coverage.sh" 877: 2024-09-05 16:27:34,773 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed ... 1609: TestDivideByZeroEndpoint 1610: test_code: | 1611: func TestDivideByZeroEndpoint(t *testing.T) { 1612: router := SetupRouter() 1613: w := httptest.NewRecorder() 1614: req, _ := http.NewRequest("GET", "/divide/10/0", nil) 1615: router.ServeHTTP(w, req) 1616: assert.Equal(t, http.StatusBadRequest, w.Code) 1617: assert.JSONEq(t, `{"error": "Cannot divide by zero"}`, w.Body.String()) ... 1707: TestSqrtNegativeNumberEndpoint 1708: test_code: | 1709: func TestSqrtNegativeNumberEndpoint(t *testing.T) { 1710: router := SetupRouter() 1711: w := httptest.NewRecorder() 1712: req, _ := http.NewRequest("GET", "/sqrt/-4", nil) 1713: router.ServeHTTP(w, req) 1714: assert.Equal(t, http.StatusBadRequest, w.Code) 1715: assert.JSONEq(t, `{"error": "Cannot take square root of a negative number"}`, w.Body.String()) ... 2198: new_imports_code: | 2199: "" 2200: test_tags: edge case 2201: 2024-09-05 16:32:20,671 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "mvn verify" 2202: 2024-09-05 16:32:28,683 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 80.0% 2203: 2024-09-05 16:32:28,692 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "mvn verify" 2204: 2024-09-05 16:32:36,306 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 100.0% 2205: 2024-09-05 16:32:36,311 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "mvn verify" 2206: 2024-09-05 16:32:43,633 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed ... 2478: expect(profileElement.innerHTML).toBe(''); 2479: }); 2480: new_imports_code: | 2481: "" 2482: test_tags: happy path 2483: 2024-09-05 16:33:35,267 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "npm run test:coverage" 2484: 2024-09-05 16:33:36,621 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 48.86% 2485: 2024-09-05 16:33:36,630 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "npm run test:coverage" 2486: 2024-09-05 16:33:37,904 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed ... 2620: 2024-09-05 16:34:03,517 - cover_agent.CoverAgent - INFO - Desired Coverage: 70% 2621: relevant_line_number_to_insert_imports_after: 5 2622: Streaming results from LLM model... 2623: language: python 2624: existing_test_function_signature: | 2625: def test_root(): 2626: new_tests: 2627: - test_behavior: | 2628: Test division by zero error handling 2629: lines_to_cover: | 2630: [75, 76, 77] 2631: test_name: | 2632: test_divide_by_zero_error 2633: test_code: | 2634: def test_divide_by_zero_error(): 2635: response = client.get("/divide/10/0") 2636: assert response.status_code == 400 2637: assert response.json() == {"detail": "Cannot divide by zero"} 2638: new_imports_code: | 2639: "from fastapi import HTTPException" 2640: test_tags: edge case 2641: - test_behavior: | 2642: Test square root of negative number error handling 2643: lines_to_cover: | 2644: [93, 94, 95] 2645: test_name: | 2646: test_sqrt_negative_number_error 2647: test_code: | 2648: def test_sqrt_negative_number_error(): ... 2666: new_imports_code: | 2667: "" 2668: test_tags: happy path 2669: 2024-09-05 16:34:08,484 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "pytest --cov=. --cov-report=xml --cov-report=term" 2670: 2024-09-05 16:34:09,644 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 65.12% 2671: 2024-09-05 16:34:09,654 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "pytest --cov=. --cov-report=xml --cov-report=term" 2672: 2024-09-05 16:34:10,815 - cover_agent.UnitTestGenerator - INFO - Test passed and coverage increased. Current coverage: 69.77% 2673: 2024-09-05 16:34:10,819 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "pytest --cov=. --cov-report=xml --cov-report=term" 2674: 2024-09-05 16:34:12,413 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 2675: ERROR:root:Error message: 2676: ... 2677: """ 2678: Return the square root of a number. Returns an error for negative numbers. 2679: """ 2680: if number < 0: 2681: raise HTTPException( 2682: status_code=400, detail="Cannot take square root of a negative number" 2683: ) 2684: > return {"result": math.sqrt(number)} 2685: E NameError: name 'math' is not defined 2686: app.py:97: NameError ... 2694: Coverage XML written to file coverage.xml 2695: 2024-09-05 16:34:12,417 - cover_agent.UnitTestGenerator - INFO - Running build/test command to generate coverage report: "pytest --cov=. --cov-report=xml --cov-report=term" 2696: 2024-09-05 16:34:13,583 - cover_agent.CoverAgent - INFO - Current Coverage: 69.77% 2697: 2024-09-05 16:34:13,583 - cover_agent.CoverAgent - INFO - Desired Coverage: 70% 2698: ``` 2699: Streaming results from LLM model... 2700: language: python 2701: existing_test_function_signature: | 2702: def test_divide_by_zero_error(): ... 3257: - test_behavior: | 3258: Test division by zero 3259: test_name: | 3260: test_division_by_zero 3261: test_code: | 3262: get '/divide/8/0' 3263: assert_equal 400, last_response.status 3264: assert_equal 'application/json', last_response.content_type 3265: assert_equal({ error: "Cannot divide by zero" }.to_json, last_response.body) ... 3293: - test_behavior: | 3294: Test square root of a negative number 3295: test_name: | 3296: test_sqrt_negative 3297: test_code: | 3298: get '/sqrt/-4' 3299: assert_equal 400, last_response.status 3300: assert_equal 'application/json', last_response.content_type 3301: assert_equal({ error: "Cannot take square root of a negative number" }.to_json, last_response.body) ... 3349: get '/echo/HelloWorld' 3350: assert last_response.ok? 3351: assert_equal 'application/json', last_response.content_type 3352: assert_equal({ message: "HelloWorld" }.to_json, last_response.body) 3353: new_imports_code: | 3354: "" 3355: test_tags: happy path 3356: 2024-09-05 16:36:44,970 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3357: 2024-09-05 16:36:45,247 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3358: 2024-09-05 16:36:45,259 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3359: 2024-09-05 16:36:45,537 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3360: 2024-09-05 16:36:45,541 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3361: 2024-09-05 16:36:45,820 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3362: 2024-09-05 16:36:45,824 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3363: 2024-09-05 16:36:46,105 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3364: 2024-09-05 16:36:46,108 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3365: 2024-09-05 16:36:46,388 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3366: 2024-09-05 16:36:46,391 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3367: 2024-09-05 16:36:46,671 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3368: 2024-09-05 16:36:46,675 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3369: 2024-09-05 16:36:46,955 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3370: 2024-09-05 16:36:46,958 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3371: 2024-09-05 16:36:47,237 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3372: 2024-09-05 16:36:47,241 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3373: 2024-09-05 16:36:47,520 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3374: 2024-09-05 16:36:47,524 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3375: 2024-09-05 16:36:47,804 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3376: 2024-09-05 16:36:47,808 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3377: 2024-09-05 16:36:48,084 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3378: 2024-09-05 16:36:48,087 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3379: 2024-09-05 16:36:48,363 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed ... 3431: get '/divide/8/2' 3432: assert last_response.ok? 3433: assert_equal 'application/json', last_response.content_type 3434: assert_equal({ result: 4.0 }.to_json, last_response.body) 3435: new_imports_code: | 3436: "" 3437: test_tags: happy path 3438: 2024-09-05 16:36:54,052 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3439: 2024-09-05 16:36:54,330 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3440: 2024-09-05 16:36:54,334 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3441: 2024-09-05 16:36:54,613 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3442: 2024-09-05 16:36:54,616 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3443: 2024-09-05 16:36:54,893 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3444: 2024-09-05 16:36:54,896 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3445: 2024-09-05 16:36:55,178 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed ... 3497: get '/divide/8/2' 3498: assert last_response.ok? 3499: assert_equal 'application/json', last_response.content_type 3500: assert_equal({ result: 4.0 }.to_json, last_response.body) 3501: new_imports_code: | 3502: "" 3503: test_tags: happy path 3504: 2024-09-05 16:36:59,909 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3505: 2024-09-05 16:37:00,189 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3506: 2024-09-05 16:37:00,192 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3507: 2024-09-05 16:37:00,479 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3508: 2024-09-05 16:37:00,482 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3509: 2024-09-05 16:37:00,770 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3510: 2024-09-05 16:37:00,775 - cover_agent.UnitTestGenerator - INFO - Running test with the following command: "ruby test_app.rb" 3511: 2024-09-05 16:37:01,056 - cover_agent.UnitTestGenerator - INFO - Skipping a generated test that failed 3512: 2024-09-05 16:37:01,059 - cover_agent.UnitTestGenerator - INFO - Running build/test command to generate coverage report: "ruby test_app.rb" 3513: 2024-09-05 16:37:01,355 - cover_agent.CoverAgent - ERROR - Reached maximum iteration limit without achieving desired coverage. Current Coverage: 37.25% ... 3515: + cleanup 3516: Cleaning up... 3517: + '[' -n 29c495fc8e5b81dfd64122833d8960355b9ddcc318ca1430db8a69cea0dd66a4 ] 3518: + echo 'Cleaning up...' 3519: + docker stop 29c495fc8e5b81dfd64122833d8960355b9ddcc318ca1430db8a69cea0dd66a4 3520: 29c495fc8e5b81dfd64122833d8960355b9ddcc318ca1430db8a69cea0dd66a4 3521: + docker rm 29c495fc8e5b81dfd64122833d8960355b9ddcc318ca1430db8a69cea0dd66a4 3522: 29c495fc8e5b81dfd64122833d8960355b9ddcc318ca1430db8a69cea0dd66a4 3523: ##[error]Process completed with exit code 2. ```

✨ CI feedback usage guide:

The CI feedback tool (`/checks)` automatically triggers when a PR has a failed check. The tool analyzes the failed checks and provides several feedbacks: - Failed stage - Failed test name - Failure summary - Relevant error logs In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR: ``` /checks "https://github.com/{repo_name}/actions/runs/{run_number}/job/{job_number}" ``` where `{repo_name}` is the name of the repository, `{run_number}` is the run number of the failed check, and `{job_number}` is the job number of the failed check. #### Configuration options - `enable_auto_checks_feedback` - if set to true, the tool will automatically provide feedback when a check is failed. Default is true. - `excluded_checks_list` - a list of checks to exclude from the feedback, for example: ["check1", "check2"]. Default is an empty list. - `enable_help_text` - if set to true, the tool will provide a help message with the feedback. Default is true. - `persistent_comment` - if set to true, the tool will overwrite a previous checks comment with the new feedback. Default is true. - `final_update_message` - if `persistent_comment` is true and updating a previous checks message, the tool will also create a new message: "Persistent checks updated to latest commit". Default is true. See more information about the `checks` tool in the [docs](https://pr-agent-docs.codium.ai/tools/ci_feedback/).

EmbeddedDevops1 commented 2 months ago

/describe

codiumai-pr-agent-pro[bot] commented 2 months ago

PR Description updated to latest commit (https://github.com/Codium-ai/cover-agent/commit/ac186c22baeba99dae57ab10eccbf0cded88bc6f)

[ ] Copy walkthrough table to "Files Changed" Tab

EmbeddedDevops1 commented 2 months ago

⌛ Running integration tests: https://github.com/Codium-ai/cover-agent/actions/runs/10713098491

EmbeddedDevops1 commented 2 months ago

⏳ Re-running integration tests: https://github.com/Codium-ai/cover-agent/actions/runs/10724314300

Codium-ai / cover-agent