se2p / pynguin

The PYthoN General UnIt Test geNerator is a test-generation tool for Python
https://www.pynguin.eu
MIT License
1.24k stars 75 forks source link

The type information from type hints is not be used or from the type4py in the module under test. #54

Closed TaicaiChen closed 8 months ago

TaicaiChen commented 9 months ago

Describe the bug A clear and concise description of what the bug is. When running the example.py in the quick start guide, although the example provides type annotation information for input and output data (input is int, output is str), the generated unit test code still includes test cases with non-int inputs. And when using type4py to obtain type information, it also does not take effect. To Reproduce Steps to reproduce the behaviour:

  1. Use Pynguin version '0.35.0'
  2. Use the following (minimal) code as a subject for test generation: './docs/source/_static/example.py'
  3. Use the following command line arguments to Pynguin: 'pynguin --project-path ./docs/source/_static/ --output-path ../tmp/pynguin-results --module-name example -v'

Expected behavior All generated unit test cases have input types as int, just as shown in the official documentation.

Screenshots The generated unit test cases: 20231227205028 The expected generated unit test cases: 20231227205250

Software Version (please complete the following information):

stephanlukasczyk commented 8 months ago

Pynguin's test generation algorithms are based on probabilistic decisions. Thus, the resulting test cases can differ between tool executions. In order to mitigate this, one can set a seed for the involved random-number generator (CLI option --seed). As a result, the generated output should be the same every time one executes Pynguin with the same seed value.

Independent of the annotated type information, Pynguin might (on random basis) decide to also try to use other types for the parameter values to increase the coverage even further. Consider an example function:

def foo(a: int):
    if isinstance(a, str):
        raise ValueError()

Such a guarding type check is something that we've seen in many Python functions. However, to cover the raise ValueError() statement, one needs to deliberately call foo with an object of a non-annotated type (str instead of int in the example). In order to be able to do so, Pynguin chooses also non-annotated types with some probability.