Figure out why a cut-off in similarity models decreases both recall and precision (hashtag858)

[vikramsubramanian]

In theory it should not decrease precision. )

[mayil-ai[bot]]

Possible Solution

• Review the set_cutoff function implementation to ensure that the similarity cut-off is applied correctly without affecting precision negatively.
• Ensure that the calculate_precision function correctly calculates the number of true positives and false positives after the cut-off is applied.
• Verify that the cut-off value is not set too high, which might exclude relevant items, thus decreasing precision. Adjust the threshold if necessary.
• Check if the calculate_recall function correctly calculates the number of true positives and false negatives, considering the cut-off.
• If the cut-off is configurable, consider providing a way to adjust it dynamically based on the impact on recall and precision, possibly through a configuration file or parameter.
• Ensure that the similarity scores are sorted (if necessary) before applying the cut-off, so that only the most relevant items are considered.
• Look for any hardcoded values or assumptions in the code that might lead to an incorrect application of the cut-off, such as the importance_cutoff in train method or the MIN_SAMPLE in the regressor.py file, and correct them if they are contributing to the issue.
• If there are unit tests related to the cut-off, recall, and precision calculations, review them to ensure they cover edge cases and the issue at hand.

Code snippets to check

bugbug → models → regressor.py

This snippet contains logic for calculating regression rates and printing out statistics, which could be directly related to the precision and recall issue described. https://github.com/Mayil-AI/bugbug-21dec23/blob/0acd00da46afbb37bd047c0bce06ce7cfad21568/bugbug/models/regressor.py#L304-L382 This snippet includes logic for classifying commits as causing regressions or not, which could impact the precision and recall of the model. https://github.com/Mayil-AI/bugbug-21dec23/blob/0acd00da46afbb37bd047c0bce06ce7cfad21568/bugbug/models/regressor.py#L157-L382

scripts → regressor_finder.py

This snippet is part of the regressor finder script and deals with the evaluation of found regressors, which could influence the precision and recall of the model. https://github.com/Mayil-AI/bugbug-21dec23/blob/0acd00da46afbb37bd047c0bce06ce7cfad21568/scripts/regressor_finder.py#L569-L673

bugbug → models → regression.py

This snippet is from the RegressionModel and includes logic for labeling bugs as regressions, which could affect the precision of the model. https://github.com/Mayil-AI/bugbug-21dec23/blob/0acd00da46afbb37bd047c0bce06ce7cfad21568/bugbug/models/regression.py#L1-L50