search

whitesource / unified-agent-distribution

51 stars 48 forks source link

Slow npm dependency resolution since 21.6.3 #16

Closed medianick closed 2 years ago

medianick commented 2 years ago

With the 21.6.2.2 release of the Unified Agent, our WhiteSource scan of the npm dependencies takes ~20 seconds:

[2021-08-16T14:55:23.984Z] [INFO] [2021-08-16 14:55:23,873 +0000] - Trying to resolve NPM dependencies
[2021-08-16T14:55:24.643Z] [INFO] [2021-08-16 14:55:24,545 +0000] - topFolder = /****/client
[2021-08-16T14:55:47.108Z] [INFO] [2021-08-16 14:55:46,552 +0000] - Trying to resolve NUGET dependencies

However, from 21.6.3 onward (and just tested against 21.7.2), this same step, scanning the same source code with the same configuration file, takes 20+ minutes.

Did 21.6.3 introduce a behavior that can be opted out of via config, or is this simply a bug?

LenaKleyner commented 2 years ago

Hi @medianick , the increase in the scan time of NPM projects was actually a result of a bug fix. The Unified Agent identifies open source dependencies by their SHA-1 checksum. When an SHA-1 checksum of a dependency is not found, the Unified Agent will try to retrieve it during the scan. There was an issue with the described flow which was resolved. A possible implication of this fix is an increase in the scan time, due to the SHA-1 retrieval process. However, we are currently working on optimizing this flow which will decrease the scan time. We are sorry for the inconvenience, the optimization is planned to be introduced in the coming releases, we will update once it is out.

Thanks,

Lena

medianick commented 2 years ago

Thanks, @LenaKleyner. We'll leave our tooling pinned to 21.6.2.2 until the optimized version is released.

LenaKleyner commented 2 years ago

Hi @medianick , the NPM scanning optimization discussed is introduced in the latest Unified Agent version 21.8.1 that was just released. It is expected to have a positive impact on the scan time. Please let us know if this indeed resolves your issue.

Thanks,

Lena

nabeelsaabna commented 2 years ago

Hi @medianick, we are closing this. Feel free to reopen if the issue persists. Thanks

medianick commented 2 years ago

Hi @LenaKleyner and @nabeelsaabna -- sorry for the delay in getting back to this. I have just tested with 21.9.1 (the latest as of today) and can confirm that the scan time is once again speedy; in the example I had previously provided, the timing between the Trying to resolve NPM dependencies step and the next one is once again mere seconds. The entire scan takes only about 90 seconds from start to finish, so I would consider this entirely resolved.

Thanks for the fix!