search

Cloud-Code-AI / kaizen

Seamless Bug Detection Throughout Your Development Cycle
https://cloudcode.ai/kaizen/docs/
MIT License
250 stars 32 forks source link

BUG: Failing unittest examples #388

Open MashyBasker opened 1 month ago

MashyBasker commented 1 month ago

Running the example: examples/unittest/main.py returns the following output

.kaizen/unit_test/kaizen/helpers/test_create_test_files.py
.kaizen/unit_test/kaizen/helpers/test_get_parent_folder.py
.kaizen/unit_test/kaizen/helpers/test_create_folder.py
.kaizen/unit_test/kaizen/helpers/test_create_pr_description.py
.kaizen/unit_test/kaizen/helpers/test_get_web_html.py
[sudo] password for okabe: 
.kaizen/unit_test/kaizen/helpers/test_create_pr_review_text.py
Results for ./.kaizen/unit_test/kaizen/helpers/test_create_test_files.py:
  Tests run: 7
  Failures: 6
  Errors: 0

Results for ./.kaizen/unit_test/kaizen/helpers/test_get_parent_folder.py:
  Tests run: 7
  Failures: 0
  Errors: 0

Results for ./.kaizen/unit_test/kaizen/helpers/test_create_folder.py:
  Tests run: 4
  Failures: 0
  Errors: 0

Results for ./.kaizen/unit_test/kaizen/helpers/test_create_pr_description.py:
  Tests run: 10
  Failures: 10
  Errors: 0

Results for ./.kaizen/unit_test/kaizen/helpers/test_get_web_html.py:
  Tests run: 16
  Failures: 16
  Errors: 0

Results for ./.kaizen/unit_test/kaizen/helpers/test_create_pr_review_text.py:
  Tests run: 10
  Failures: 8
  Errors: 0

Results for ./.kaizen/unit_test/kaizen/helpers/__pycache__/test_create_pr_review_text.cpython-312-pytest-8.3.2.pyc:
  Error: Unsupported file type: .pyc

Results for ./.kaizen/unit_test/kaizen/helpers/__pycache__/test_get_parent_folder.cpython-312-pytest-8.3.2.pyc:
  Error: Unsupported file type: .pyc

Results for ./.kaizen/unit_test/kaizen/helpers/__pycache__/test_get_web_html.cpython-312-pytest-8.3.2.pyc:
  Error: Unsupported file type: .pyc

Results for ./.kaizen/unit_test/kaizen/helpers/__pycache__/test_create_pr_description.cpython-312-pytest-8.3.2.pyc:
  Error: Unsupported file type: .pyc

Results for ./.kaizen/unit_test/kaizen/helpers/__pycache__/test_create_folder.cpython-312-pytest-8.3.2.pyc:
  Error: Unsupported file type: .pyc

Results for ./.kaizen/unit_test/kaizen/helpers/__pycache__/test_create_test_files.cpython-312-pytest-8.3.2.pyc:
  Error: Unsupported file type: .pyc

My config.json

{
    "language_model": {
        "provider": "litellm",
        "enable_observability_logging": false,
        "redis_enabled": true,
        "models": [
            {
                "model_name": "default",
                "litellm_params": {
                    "model": "groq/llama3-8b-8192"
                }
            }
        ]
    },
    "github_app": {
        "check_signature": false,
        "auto_pr_review": true,
        "edit_pr_desc": true,
        "process_on_push": true,
        "auto_unit_test_generation": false
    }
}
sauravpanda commented 1 month ago

We can ignore the error on .pyc as it's irrelevant. It seems like it's not an error, but the test cases are failing. Failing means either our code isn't covering the edge cases or the test cases are not properly testing the edge cases. ( I will work on it a little tomorrow)

We must look at the code and see why and how to fix it. Heads up: The unit test case generation isn't perfect yet; we are iterating on it.

I think if you can try to generate the test case for the rust file (you can pass the dir of the rust file and try to see the test cases generated along with their execution.

MashyBasker commented 1 month ago

Sure. I'll also try to log the assertion messages of the failing tests to get some idea.

sauravpanda commented 1 month ago

That sounds great; feel free to share your findings here or on Discord!