microsoft / component-detection

Scans your project to determine what components you use
MIT License
412 stars 87 forks source link

Make Python detection more resilient to unexpected failure cases #962

Closed cobya closed 7 months ago

cobya commented 7 months ago

In the Python detectors, there are a few exceptions which are currently unhandled, leading to failures in detection which are not well presented to the user. Add some additional handling when this occurs so users have a good idea of why the error occurred and can investigate themselves to see if detection or their requirements need changes. Related issue #928

First case: Python package has multiple dependencies specified with the same package name, but differing version constraints. An exception will be hit because of the ToDictionary call which expects distinct values.

System.ArgumentException: An item with the same key has already been added. Key: portalocker
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at Microsoft.ComponentDetection.Detectors.Pip.PythonResolver.InvalidateAndReprocessAsync(PythonResolverState state, PipGraphNode node, PipDependencySpecification newSpec)
   at Microsoft.ComponentDetection.Detectors.Pip.PythonResolver.ProcessQueueAsync(ISingleFileComponentRecorder singleFileComponentRecorder, PythonResolverState state)
   at Microsoft.ComponentDetection.Detectors.Pip.PythonResolver.ResolveRootsAsync(ISingleFileComponentRecorder singleFileComponentRecorder, IList`1 initialPackages)
   at Microsoft.ComponentDetection.Detectors.Pip.PipComponentDetector.OnFileFoundAsync(ProcessRequest processRequest, IDictionary`2 detectorArgs)

I'll be filing an issue here for further investigation, but for now, choose the latest specifier given to avoid skipping the dependency all together.

Second case: Python package is given version constraints either from requirements.txt or package dependency version specifiers which are not resolved in PyPi. Previously only the package name was outputted to the user, meaning they don't know why a valid version wasn't found if the package exists on PyPi.

codecov[bot] commented 7 months ago

Codecov Report

Attention: 38 lines in your changes are missing coverage. Please review.

Comparison is base (b65fa8a) 75.5% compared to head (34531b8) 74.9%.

Files Patch % Lines
...onentDetection.Detectors/pip/PythonResolverBase.cs 39.2% 30 Missing and 4 partials :warning:
...ft.ComponentDetection.Detectors/pip/IPyPiClient.cs 33.3% 2 Missing :warning:
...ComponentDetection.Detectors/pip/PythonResolver.cs 77.7% 0 Missing and 2 partials :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #962 +/- ## ======================================= - Coverage 75.5% 74.9% -0.7% ======================================= Files 234 235 +1 Lines 10070 10056 -14 Branches 993 986 -7 ======================================= - Hits 7607 7533 -74 - Misses 2190 2251 +61 + Partials 273 272 -1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

cobya commented 7 months ago

Related issue for further investigation #963

github-actions[bot] commented 7 months ago

👋 Hi! It looks like you modified some files in the Detectors folder. You may need to bump the detector versions if any of the following scenarios apply:

If none of the above scenarios apply, feel free to ignore this comment 🙂