microsoft / component-detection

Scans your project to determine what components you use
MIT License
396 stars 81 forks source link

Enable parallelism by detector #1171

Closed pauld-msft closed 2 weeks ago

pauld-msft commented 2 weeks ago

Adds some configurable parallelism to a single detector, e.g. within PipReport.

Added some simple pip reports that take 3-5 seconds to generate.

Running with the MaxDetectionThreads set to 1: image

Running without the MaxDetectionThreads (default of 10): image

Finally, with MaxDetectionThreads set to 2: image

Scan log from the 2 parallel threads is also interesting since you can see that 2 pip reports are triggered at the same time and the next set is triggered once these finish: GovCompDisc_Log_20240612152920140_41928.log

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 96.29630% with 1 line in your changes missing coverage. Please review.

Project coverage is 76.3%. Comparing base (22a88b5) to head (a694d82).

Files Patch % Lines
...Orchestrator/Services/DetectorProcessingService.cs 66.6% 0 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1171 +/- ## ======================================= + Coverage 75.6% 76.3% +0.7% ======================================= Files 257 257 Lines 11481 11495 +14 Branches 1154 1156 +2 ======================================= + Hits 8680 8772 +92 + Misses 2460 2384 -76 + Partials 341 339 -2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

cobya commented 2 weeks ago

Looks like there are some parallelization issues with the GoWithReplace detector - can you take a look?

github-actions[bot] commented 2 weeks ago

👋 Hi! It looks like you modified some files in the Detectors folder. You may need to bump the detector versions if any of the following scenarios apply:

If none of the above scenarios apply, feel free to ignore this comment 🙂

pauld-msft commented 2 weeks ago

Looks like there are some parallelization issues with the GoWithReplace detector - can you take a look?

@cobya Yep, seems like go w/ replace was using a class level list of components, which was re-used by the different threads. Changed this up to initialize the component list as the file found method is invoked, which should make it thread safe. Also did a quick spot check and could not find similar issues with the other detectors