Test the tracer on five projects

[RecurvedBow]

• Test the project on five projects (use #9 to find good repositories)
• Analyze and benchmark test the performance (speed, memory, accuracy)

[RecurvedBow]

Tested the poetry repository (https://github.com/python-poetry/poetry.git). The goal is to find out how high the completeness and correctness is and find out why the values are low if that is the case. Used configuration:

[pytypes]
project = "poetry"
stdlib_path = "/usr/lib/python3.10"
proj_path = "/home/secret/PycharmProjects/ProjectsToTrace/poetry/poetry"
venv_path = "/home/secret/.cache/pypoetry/virtualenvs/poetry-1R9kiBM8-py3.10"
benchmark_performance = true

[[unifier]]
name = "dupl"
kind = "dedup"

[[unifier]]
name = "test"
kind = "drop_test"
test_name_pat = "test_"

[[unifier]]
name = "mult2"
kind = "drop_mult_var"

[[unifier]]
name = "mult3"
kind = "drop_mult_var"
min_amount_types_to_drop = 3

[[unifier]]
name = "st_r"
kind = "unify_subty"

[[unifier]]
name = "st_sr"
kind = "unify_subty"
only_unify_if_base_was_traced = true

[[unifier]]
name = "first"
kind = "keep_only_first"

[[unifier]]
name = "mt"
kind = "drop_min_threshold"
min_threshold = 0.3

[[unifier]]
name = "unify"
kind = "union"

python3.10 main.py fetch -u https://github.com/python-poetry/poetry.git -o ../ProjectsToTrace/poetry -eCreated the corresponding environment, installed the missing modules from this project, executed the tests with pytest and after the tracing, reset all file changes so that the traced repository matches the original. python3.10 main.py typegen -p ../ProjectsToTrace/poetry/poetry/ -u test -u mt -u dupl -u mult2 -u dupl -u first -g eval_inline python3.10 main.py evaluate -o ../ProjectsToTrace/poetry/Original/ -t ../ProjectsToTrace/poetry/poetry/ -s ./evaluation_data/ -d poetry Then, the evaluation template is used to evaluate the tracing results. These are the following insights:

• Test files are overwritten as there are classes & functions which do not start with test_ . It might be useful to drop data about test files.

• By checking out the files with 0% completeness, it has been found that the parameters in multi-line function definitions are not hinted despite the local variables having been hinted.

• The original files in many cases do not have type hints about local variables.

• If a test fails, there isn't any trace data for the tested objects/functions, lowering the completeness.

• A bug has been found: If a[b] = 1, then the generated file has a:int = 1, removing the [b].

• The correctness is lowered due to replacing the abstract type hint with the subclass type hint. For example: Path changes to PosixPath. While the traced type hint is correct, it differs from the original type hint. Thanks to the insights collected from the evaluation, added #202, #203, #204 .

The evaluation regarding performance data shows that linear regression isn't possible. The data is too far spread apart. However, the data also shows that on average, the standard tracer is faster than the optimized tracer. Thus, it is recommended to use the standard tracer instead of the optimized tracer.

The evaluation regarding performance data shows that linear regression isn't possible. The data is too far spread apart. However, the data also shows that on average, the standard tracer is faster than the optimized tracer. Thus, it is recommended to use the standard tracer instead of the optimized tracer #205. Due to being unable to upload the .zip here, it is accessible in the drive https://drive.google.com/drive/folders/13_ItcpjVTY0qW4wwYeKLeEf1LqOqbQb8?usp=sharing.

Due to time constraints, only this repository could be tested and fully evaluated.

[RecurvedBow]

Small update: Traced the poetry repository with the current state of the project (commit 1dadab04cb62f5bbcab973c20fcea1d6516193c8). Same configuration & commands as before, but changed used unifiers to: python3.10 main.py typegen -p ../ProjectsToTrace/poetry/poetry/ -u test -u mt -u dupl -u mult3 -u unify -u mult2 -u first -g eval_inline

However, evaluation results are still very similar. The same insights still apply. It is accessible in the drive https://drive.google.com/drive/folders/13_ItcpjVTY0qW4wwYeKLeEf1LqOqbQb8?usp=sharing. Updated the final presentation with these results.