search

jeremylong / DependencyCheck

OWASP dependency-check is a software composition analysis utility that detects publicly disclosed vulnerabilities in application dependencies.
https://owasp.org/www-project-dependency-check/
Apache License 2.0
6.17k stars 1.25k forks source link

Downloads after NVD init don't seem to be using proxy settings correctly #6800

Open DocMoebiuz opened 2 weeks ago

DocMoebiuz commented 2 weeks ago

Describe the bug We have a corporate proxy in place and I am providing the settings including proxy user and proxy pass through the JAVA_TOOL_OPTIONS as described in the documentation.

This works for the NVD updates, but as soon as I get to the the point where it wants to init the retireJS repo or download the publishedSupressions.xml, then I receive a 407 error from the proxy.

Version of dependency-check used The problem occurs using version 10.0.1

Log file

[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 240 minutes
[ERROR] Failed to initialize the RetireJS repo
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to initialize the RetireJS repo
        at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:152)
        at org.owasp.dependencycheck.data.update.RetireJSDataSource.update(RetireJSDataSource.java:95)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json' to 'C:\actions-runner\_work\NSDT\dependency-check\data\jsrepository.json'; Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:100)
        at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:150)
        ... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
        ... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
        ... 11 common frames omitted
[WARN] Failed to update hosted suppressions file, results may contain false positives already resolved by the DependencyCheck project
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to update the hosted suppressions file
        at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:156)
        at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.update(HostedSuppressionsDataSource.java:87)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml' to 'C:\actions-runner\_work\NSDT\dependency-check\data\publishedSuppressions.xml'; Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:83)
        at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:154)
        ... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
        at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
        ... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
        ... 11 common frames omitted
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
org.owasp.dependencycheck.data.update.exception.UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
        at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update(KnownExploitedDataSource.java:93)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
        at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
        at org.owasp.dependencycheck.App.runScan(App.java:262)
        at org.owasp.dependencycheck.App.run(App.java:194)
        at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
        at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update(KnownExploitedDataSource.java:80)
        ... 6 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
        ... 8 common frames omitted
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.
[ERROR] One or more fatal errors occurred
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR] No documents exist

To Reproduce Steps to reproduce the behavior:

  1. Run dependency check with proxy parameters provided through JAVA_TOOLS_OPTIONS
  2. Wait for the NVD to finish
  3. See error

Expected behavior The process should download additional resources without error using the same proxy config.

Additional context I have successfully downloaded the files manually, so it is not the proxy that blocks these specific URLs

chadlwilson commented 2 weeks ago

The correct env var is JAVA_TOOL_OPTIONS not JAVA_TOOLS_OPTIONS. Are you sure you've configured it correctly?

Your log above shows the below, so it never tried to call NVD.

[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 240 minutes
DocMoebiuz commented 2 weeks ago

It was a typo - I am using JAVA_TOOL_OPTIONS NVD is skipped because it was downloaded less than 4 hours ago. That's what I am saying, the NVD download works... the downloads thereafter dont.

chadlwilson commented 2 weeks ago

Well the error implies it is talking to your proxy, hence getting Proxy returns "HTTP/1.1 407 Proxy Authentication Required" so folks would probably need to see the specific arguments you are sending (redacted) and how you are sending them (Gradle? Maven? Standalone? did it used to work but no longer does? can you force it to update NVD and show a log which shows NVD working but other data sources not?).

Depending on your proxy configuration, there are many reasons this could happen, e.g the proxy happens to whitelist or allow through the NVD API host, but not cisa.gov or github.io etc. Would need to see the args and a more complete log showing the NVD API working to tell.

DocMoebiuz commented 2 weeks ago

i can download from cisa.gov or github.io with the same proxy settings from the command line, so this rules out that the proxy is having some specific issues with these URLs.

chadlwilson commented 2 weeks ago

OK, that's useful info, but folks would still need to know what you're supplying to ODC and which specific variant you are running.

DocMoebiuz commented 2 weeks ago

i am not sure what you mean with "supplying to ODC"? - I have a dotnet project and I am trying to run the dependency-check.bat CLI on windows. I followed the CLI documentation which explains how to pass in the proxy relevant settings via the JAVA_TOOL_OPTIONS.

It seems to me as if the NVD download requests are using the proxy information, whereas all other requests don't. I have to run the CLI cmd with --disableRetireJS --disableKnownExploited so that I get passed those.

chadlwilson commented 2 weeks ago

What is the specific value of JAVA_TOOL_OPTIONS you are using and how are you exposing it to the CLI? What commands are you using? How can someone else replicate your problem reliably without having to guess what you are doing and replicate your precise proxy or runtime environment (which is not disclosed)?

While you may be right that a core proxy functionality is fundamentally broken, given the very wide user base of ODC it seems more likely that you’re doing something wrong, there is something specific to your environment that is wrong - or the documentation is misleading.

If you don’t explain precisely what you’re doing or include logs that show what you say is happening (NVD API working), we can’t easily get anywhere.

DocMoebiuz commented 2 weeks ago

My JAVA_TOOL_OPTIONS are like this, and set as environment variables:

-Dhttps.proxyHost=${{ vars.PROXY_HOST }} -Dhttps.proxyPort=${{ vars.PROXY_PORT }} -Dhttps.proxyUser=${{ secrets.PROXY_USER }} -Dhttps.proxyPassword=${{ secrets.PROXY_PASS }}

The CLI tool is started like this: ..\dependency-check\bin\dependency-check.bat --scan .\Code --format HTML --project "myproject.sln" --out .\report --nvdApiKey $env:NVD_API_KEY --disableRetireJS --disableKnownExploited

As I already stated before, the NVD downloads went fine with these settings. If I drop the --disableRetireJS or --disableKnownExploited option then I would see the error log entries from my first post.

jeremylong commented 2 weeks ago

Does this work for you (of course updating the proxy?

curl --proxy "http://user:pwd@127.0.0.1:1234" "https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json"
chadlwilson commented 2 weeks ago

As I already stated before, the NVD downloads went fine with these settings. If I drop the --disableRetireJS or --disableKnownExploited option then I would see the error log entries from my first post.

Yes, you stated/told, but you didn't show via logs. I wouldn't ask to see if I didn't think it was useful to eliminate problems or assumptions about how ODC works, this is why I asked:

can you force it to update NVD and show a log which shows NVD working but other data sources not?).

Anyway, good luck to you.

DocMoebiuz commented 2 weeks ago

Does this work for you (of course updating the proxy?

curl --proxy "http://user:pwd@127.0.0.1:1234" "https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json"

Yes, that's working, but I had to switch to another proxy url to make it work with curl whereas it was working fine with Invoke-WebRequest -Uri $url -OutFile $output -Verbose. I will now double check one more time and maybe use the --purge command to verify whether the NVD can be downloaded (as suggested by @chadlwilson)

Did I mention that I hate proxies 😄

DocMoebiuz commented 2 weeks ago

Unforunately, it's the same result like before, now also showing that the NVD downloads work properly:

4:47:38,161 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
14:47:38,161 |-INFO in ch.qos.logback.classic.joran.action.ContextNameAction - Setting logger context name as [dependency-check]
14:47:38,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
14:47:38,161 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [console]
14:47:38,[17](actions/runs/6374661/job/18382986#step:6:18)7 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.commons.jcs] to ERROR
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.hc] to ERROR
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
14:47:38,192 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[ROOT]
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
14:47:38,192 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@5bb21b69 - Registering current configuration as safe fallback point

[INFO] Checking for updates
[INFO] NVD API has 28 records in this update
[INFO] Downloaded 28/28 (100%)
[INFO] Completed processing batch 1/1 (100%) in 233ms
Error:  Failed to initialize the RetireJS repo
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to initialize the RetireJS repo
    at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:152)
    at org.owasp.dependencycheck.data.update.RetireJSDataSource.update(RetireJSDataSource.java:95)
    at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
    at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
    at org.owasp.dependencycheck.App.runScan(App.java:262)
    at org.owasp.dependencycheck.App.run(App.java:194)
    at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json' to 'C:\actions-runner\_work\NSDT\dependency-check\data\jsrepository.json'; Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:100)
    at org.owasp.dependencycheck.data.update.RetireJSDataSource.initializeRetireJsRepo(RetireJSDataSource.java:150)
    ... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://raw.githubusercontent.com/Retirejs/retire.js/master/repository/jsrepository.json; unable to connect.
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
    ... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
    at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:[18](https://github/actions/runs/6374661/job/18382986#step:6:19)5)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:206)
    ... 11 common frames omitted
[WARN] Failed to update hosted suppressions file, results may contain false positives already resolved by the DependencyCheck project
org.owasp.dependencycheck.data.update.exception.UpdateException: Failed to update the hosted suppressions file
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:156)
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.update(HostedSuppressionsDataSource.java:87)
    at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
    at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
    at org.owasp.dependencycheck.App.runScan(App.java:262)
    at org.owasp.dependencycheck.App.run(App.java:[19](https://actions/runs/6374661/job/18382986#step:6:20)4)
    at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml' to 'C:\actions-runner\_work\NSDT\dependency-check\data\publishedSuppressions.xml'; Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:152)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:83)
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions(HostedSuppressionsDataSource.java:154)
    ... 7 common frames omitted
Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:267)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:163)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile(Downloader.java:138)
    ... 9 common frames omitted
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
    at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:[20](https://github/actions/runs/6374661/job/18382986#step:6:21)6)
    ... 11 common frames omitted
jeremylong commented 2 weeks ago

...I had to switch to another proxy url to make it work with curl

Sounds like an environment issue? possibly one of the proxies blocks access to some resources? I know some users have had to work with the networking team to allow the connections on the proxy.

DocMoebiuz commented 2 weeks ago

Well, as I wrote already: I can curl the URLs that are failing from the command line with the same options from JAVA_TOOL_OPTIONS. It can't be the proxy preventing access... it seems to me as if the NVD requests pick up the proxy settings but for the other "modules" like the RetireJS they aren't.

jeremylong commented 2 weeks ago

You can also host these files inside your network - see https://jeremylong.github.io/DependencyCheck/data/index.html

DocMoebiuz commented 1 week ago

So, looking at this page: https://jeremylong.github.io/DependencyCheck/data/proxy.html

Is it fair to assume that if JAVA_TOOL_OPTIONS is set correctly, all subsequent requests by dependency-check will use those? I am asking since I am starting dependency check in a github action which always has it's own set of env variables, etc.

Also, the page https://jeremylong.github.io/DependencyCheck/data/proxy.html mentions legacy proxy config. I am wondering if that is required if JAVA_TOOL_OPTIONS is set correctly.

jeremylong commented 1 week ago

Yes, correctly setting the JAVA_TOOL_OPTIONS should be used for all requests. The legacy options were left there to make it easier for upgrades.