GitGuardian / ggshield

Find and fix 400+ types of hardcoded secrets and 70+ types of infrastructure-as-code misconfigurations.
https://gitguardian.com
MIT License
1.62k stars 143 forks source link

Fix ggshield creating too large payloads #823

Closed fnareoh closed 8 months ago

fnareoh commented 8 months ago

Aims at fixing the issue with scanning large documents where chunks can end up being larger than the maximum server payload, first reported in #555.

Depends on a change to py-gitguardian, see PR here, to pass the server payload to the GGClient.

To facilitate the computation of chunk size an utf8_encoded_size property was added and calls a new method _read_content. Some of the code is a bit clunky with if self.content is None statement even tough we just called a read method to make sure it wouldn't be None, but this way pyright doesn't complain.

I left a constant margin in the chunk size to encode the metadata and tested it with a scan of 10 000 files of 1Kb but it might be better to have it adapt depending on the number of files in the chunk ?

codecov-commenter commented 8 months ago

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (68180d6) 91.60% compared to head (a3bab0f) 91.54%.

Files Patch % Lines
ggshield/core/scan/scannable.py 80.95% 4 Missing :warning:
ggshield/verticals/secret/secret_scanner.py 85.71% 1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #823 +/- ## ========================================== - Coverage 91.60% 91.54% -0.06% ========================================== Files 168 168 Lines 6929 6941 +12 ========================================== + Hits 6347 6354 +7 - Misses 582 587 +5 ``` | [Flag](https://app.codecov.io/gh/GitGuardian/ggshield/pull/823/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GitGuardian) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/GitGuardian/ggshield/pull/823/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GitGuardian) | `91.54% <86.11%> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GitGuardian#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.