Refactor: save lines in extended match and add censor method

fnareoh commented 1 month ago

Context

After having worked on factorizing make_matches the new goal is to start storing context lines in the Extended Match and have the output entirely depend those line and no longer the result.file.content. In particular this means a way to censor Extended Match based on the lines and the indices in MatchSpan and no longer relying on censor content.

What has been done

This MR is divided into several commits and test should pass after each commit (best reviewed by commit?).

Several important change have been made:

the Line class has been changed from a NamedTuple to a SimpleNamespace to be able to modify the line.content for censoring.
The display patch hunk indication is no longer present in the patch output as it is not really part of the file and we do not display the patch symbols '+/-/ ' . This means some test secrets are considered shorter and the censoring (proportional to the length of the secret) is shorter as well (so some test in output test text have been changed accordingly).
The '+' have been removed of multi line secret in patches as they where not present in single ligne secret nor in the first and last lines of the multiline secret.
Some test in test output text had falty Match index end creating falty output, it has been cleanned up.

Validation

Test pass and output is more coherent now ? I did my best to make it reviewable but it was quite tricky to work with...

PR check list

[ ] As much as possible, the changes include tests (unit and/or functional)
[ ] If the changes affect the end user (new feature, behavior change, bug fix) then the PR has a changelog entry (see doc/dev/getting-started.md). If the changes do not affect the end user, then the skip-changelog label has been added to the PR.

fnareoh commented 1 month ago

As discussed, I added back the display of patches hunk and unified the display of patch line "category".

The error on unit test on windows seems to have been due outdated cassettes with faulty indices I'm really surprised the other OS work (and it is a bit worrying).

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 98.00000% with 2 lines in your changes missing coverage. Please review.

Please upload report for BASE (agateau/fix-extended-match@8b5d34a). Learn more about missing BASE report.

Files	Patch %	Lines
ggshield/core/lines.py	85.71%	2 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## agateau/fix-extended-match #899 +/- ## ============================================================= Coverage ? 92.00% ============================================================= Files ? 172 Lines ? 7094 Branches ? 0 ============================================================= Hits ? 6527 Misses ? 567 Partials ? 0 ``` | [Flag](https://app.codecov.io/gh/GitGuardian/ggshield/pull/899/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GitGuardian) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/GitGuardian/ggshield/pull/899/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GitGuardian) | `92.00% <98.00%> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GitGuardian#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

salome-voltz commented 1 month ago

Minor remarks with suggestions, but feel free to merge :smile:

GitGuardian / ggshield