search

jeremylong / DependencyCheck

OWASP dependency-check is a software composition analysis utility that detects publicly disclosed vulnerabilities in application dependencies.
https://owasp.org/www-project-dependency-check/
Apache License 2.0
6.32k stars 1.26k forks source link

Version 8.0.0. KnownExploitedDataSource proxy HTTP 403 #5313

Open marwin1991 opened 1 year ago

marwin1991 commented 1 year ago

KnownExploitedDataSource Line: 78

https://github.com/jeremylong/DependencyCheck/blob/f3d806de22002d932fe12a706f4b9fa91ed3322d/core/src/main/java/org/owasp/dependencycheck/data/update/KnownExploitedDataSource.java

                //TODO - add all the proxy config, likely use the same as configured for NVD
                final HttpResourceConnection conn = new HttpResourceConnection(settings);

In class HostedSuppressionsDataSource i do not see such a commonet:

https://github.com/jeremylong/DependencyCheck/blob/f3d806de22002d932fe12a706f4b9fa91ed3322d/core/src/main/java/org/owasp/dependencycheck/data/update/HostedSuppressionsDataSource.java

Logs:

[ERROR] Failed to execute goal org.owasp:dependency-check-maven:8.0.0:check (default-cli) on project iaml: Fatal exception(s) analyzing aaaaaaaaaa: One or more exceptions occurred during analysis:
[ERROR]     UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR]         caused by DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR]         caused by IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"
[ERROR]     NoDataException: No documents exist
[ERROR] -> [Help 1]
[ERROR]
lukasT commented 1 year ago

I got a similar problem (getting 403 when trying to download the external file) which might be related, if the plugin was executed within a gitlab-ci pipeline job:

[ERROR]     UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR]         caused by DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
[ERROR]         caused by DownloadFailedException: Error retrieving https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; received response code 403; Forbidden
[ERROR]     NoDataException: No documents exist

so maybe our problem is the same that somehow the https://www.cisa.gov/ page does not allow your proxy and my direct download...not sure, why though

jeremylong commented 1 year ago

As @aikebah noted in PR #5330- the proxy configuration is already being used if it is configured.

Are there rules setup to allow the NVD that would need to be replicated in your proxy for cisa.gov?

marwin1991 commented 1 year ago

As @aikebah noted in PR #5330- the proxy configuration is already being used if it is configured.

Are there rules setup to allow the NVD that would need to be replicated in your proxy for cisa.gov?

DevOps team has changed something in the configuration to fix this, but error as mentioned before occured while executing command:

      mvn $COMPILE \
      --batch-mode \
      -Dhttp.proxyHost=X.X.X.X \
      -Dhttps.proxyHost=X.X.X.X \
      -Dhttp.proxyPort=8080 \
      -Dhttps.proxyPort=8080 \
      -Dhttp.nonProxyHosts="a.b.c" \
      -Dhttps.nonProxyHosts="*.a.b.c" \
      $SECURITY_DEPENDENCY_SCANNING_MAVEN_EXTRA_ARGS \
      org.owasp:dependency-check-maven:check -Dformat=JUNIT
aikebah commented 1 year ago

DevOps team has changed something in the configuration to fix this

My gut feel: they whitelisted the URL or the entire CISA website on your proxy

marwin1991 commented 1 year ago

Hello, what about version 8.0.1 ? It's not using proxy to get publishedSuppressions.xml

Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://jeremylong.github.io/DependencyCheck/suppressions/publishedSuppressions.xml; unable to connect.
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection (HttpResourceConnection.java:267)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch (HttpResourceConnection.java:163)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:130)
    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:74)
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.fetchHostedSuppressions (HostedSuppressionsDataSource.java:133)
    at org.owasp.dependencycheck.data.update.HostedSuppressionsDataSource.update (HostedSuppressionsDataSource.java:77)
    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:900)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:705)
    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:631)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1914)
    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:1096)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke (Method.java:578)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"
    at sun.net.www.protocol.http.HttpURLConnection.doTunneling0 (HttpURLConnection.java:2262)
aikebah commented 1 year ago

Well... still it IS using your proxy, but your proxy is returning

Proxy returns "HTTP/1.1 403 Forbidden

so you need to find out why your proxy is denying you access to the CISA known exploited vulnerabilities

aikebah commented 1 year ago

most likely you run it in a network-access restricted environment with a proxy that has whitelisted "allowed locations" and the CISA website is not among them, so contact your network team

marwin1991 commented 1 year ago

Tank you so much @aikebah

sanddorn commented 1 year ago

Is there a known bann-list on cisa.gov for the hetzner network? I get a 403 from cisa.gov on all of my hetzner servers (cloud and root-server), but not on digitalocean or on the telekom.de network via DSL.

lukasT commented 1 year ago

@sanddorn that would explain my problem too as we have gitlab runners on hetzner.

sanddorn commented 1 year ago

@lukasT You can simply check by a curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json That runs on most systems, but on Hetzner. That's part of my question. It seems, there es a ban on the hetzner networks.

On Hetzer, I get an 403, 200 and lots of data anywhere else.

sanddorn commented 1 year ago

@lukasT Changing Runner to Hillsboro (US) solves the problem.

thomseno commented 1 year ago

@lukasT You can simply check by a curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json That runs on most systems, but on Hetzner. That's part of my question. It seems, there es a ban on the hetzner networks.

On Hetzer, I get an 403, 200 and lots of data anywhere else.

Experiencing the same problem... gitlab-runner on Hetzner, scan is failing because not able to download from cisa.gov. Downgraded to maven-plugin version 7.4.4.

sblack4 commented 1 year ago

I'm also getting a 403 from a runner on the Hetzner network. I can't find anything about a ban list, though.

jeremylong commented 1 year ago

The options would be to mirror the catalog or disable the analyzer.

thomseno commented 1 year ago

The options would be to mirror the catalog or disable the analyzer.

How to disable the Known Exploited Vulnerability analyzer from Maven plugin?

thomseno commented 1 year ago

The options would be to mirror the catalog or disable the analyzer.

How to disable the Known Exploited Vulnerability analyzer from Maven plugin?

Forget it... found it: -DknownExploitedEnabled=false

aikebah commented 1 year ago

Forget it... found it: -DknownExploitedEnabled=false

Either like that, or by specifying <knownExploitedEnabled>false</knownExploitedEnabled> in the plugin configuration section in your pom-file

alan-czajkowski commented 1 year ago

you can use a mirror or proxy via the plugin config like so:

<knownExploitedUrl>https://your-mirror-or-proxy/repository/cisa-files/sites/default/files/feeds/known_exploited_vulnerabilities.json</knownExploitedUrl>
alan-czajkowski commented 1 year ago

@jeremylong you should probably update the docs to make it more obvious on how to configure this stuff

UbiNickUfer commented 1 year ago

The site is blocked in GCP europe-west too. Can't tell if it blocked the whole GCP network but this is already pretty bad on its own. Why are they doing this? Do they want that nobody is using this?

msaubier commented 1 year ago

My team has been running into this issue intermittently this week...sometimes it works, sometimes it doesn't when attempting to download the CISA feed in our AWS CodePipeline. Is there a drawback to disabling the CISA Known Exploited Vulnerability analyzer? Would we potentially miss scanning for any CVEs?

jeremylong commented 1 year ago

The known exploited vulnerability catalog does not add any new vulnerabilities... but any vulnerabilities in the catalog likely need to be patched ASAP as there are known attacks happening using the CVE.

On Thu, Jun 29, 2023, 2:52 PM msaubier @.***> wrote:

My team has been running into this issue intermittently this week...sometimes it works, sometimes it doesn't. Is there a drawback to disabling the CISA Known Exploited Vulnerability analyzer? Would we potentially miss scanning for any CVEs?

— Reply to this email directly, view it on GitHub https://github.com/jeremylong/DependencyCheck/issues/5313#issuecomment-1613635432, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGSVQQYBZFMDYPOZOMGEA3XNXFGLANCNFSM6AAAAAAT4SQELM . You are receiving this because you were mentioned.Message ID: @.***>

lgolubenkobit commented 4 months ago

This week we are getting this error 

[INFO] --- dependency-check:9.1.0:check (default) @ canvas-root ---
[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 480 minutes
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[ERROR] Error retrieving https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; received response code 403; Forbidden
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
org.owasp.dependencycheck.data.update.exception.UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
    at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update (KnownExploitedDataSource.java:93)

and after that we cannot analyze any project by some time.

We are using apiKey and this error is only reported with known_exploited_vulnerabilities.json, previous update is done ok.

If i try to do curl it works ok. curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

We had the same problem with version 7. About 1 month ago we updated dependency-check from 7 to 9 and works ok without problems on more than 30 pipelines that sometimes can run at same time, until this week that on some moments all pipelines running fail.

lgolubenkobit commented 4 months ago

This week we are getting this error

[INFO] --- dependency-check:9.1.0:check (default) @ canvas-root ---
[INFO] Checking for updates
[INFO] Skipping the NVD API Update as it was completed within the last 480 minutes
[INFO] Updating CISA Known Exploited Vulnerability list: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
[ERROR] Error retrieving https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; received response code 403; Forbidden
[ERROR] org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
org.owasp.dependencycheck.data.update.exception.UpdateException: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json; unable to connect.
    at org.owasp.dependencycheck.data.update.KnownExploitedDataSource.update (KnownExploitedDataSource.java:93)

and after that we cannot analyze any project by some time.

We are using apiKey and this error is only reported with known_exploited_vulnerabilities.json, previous update is done ok.

If i try to do curl it works ok. curl -v https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

We had the same problem with version 7. About 1 month ago we updated dependency-check from 7 to 9 and works ok without problems on more than 30 pipelines that sometimes can run at same time, until this week that on some moments all pipelines running fail.

Our pipelines run on AWS.

2024-05-13T16:15:51 First pipeline running on the day, but according last pipeline runned 4 days ago, then update and works ok. image

2024-05-13T17:27:51, about one hour later, in the same repository (using cache for .m2 folder, then nothing should be updated), but dependency list is updated and then get error image

2024-05-13T17:42:04, 10 minutes later, after pipeline retry, works ok image

There aren't other pipelines running at same time (using the same key).

aikebah commented 4 months ago

2024-05-13T17:27:51, about one hour later, in the same repository (using cache for .m2 folder, then nothing should be updated), but dependency list is updated and then get error

Default validForHours of NVD API data is 4hrs, but updates were requested, so it looks like you were using a different .m2 cache in that run. KEV is using a validity period of 24hrs by default, so again indicating that you likely got a different .m2 cache for that run.

lgolubenkobit commented 4 months ago

2024-05-13T17:27:51, about one hour later, in the same repository (using cache for .m2 folder, then nothing should be updated), but dependency list is updated and then get error

Default validForHours of NVD API data is 4hrs, but updates were requested, so it looks like you were using a different .m2 cache in that run. KEV is using a validity period of 24hrs by default, so again indicating that you likely got a different .m2 cache for that run.

Assuming it could be an error that the same .m2 directory is not being used in the same repo, why do we get the http 403 forbidden error without having exceeded the usage rate when using apiKey? image

What causes a http 403 updating CISA Known Exploited Vulnerability list?

aikebah commented 4 months ago

What causes it you should ask the CISA.

In any case the NVD apiKey is for access to NIST NVD CVE-API and has no effect or usage with the CISA KEV json downloads (they are different resources operated by different organisations)

Regarding the ways in which you could have multiple .m2 caches: that would fully depend on how your AWS cloud build pipelines environment is operated.

E.g. I can imagine setups with caches to be datacenter-, serverrack- or even buildrunner-local rather than a global share that is replicated introducing synchronization latencies or a globally shared single share introducing continuous I/O latencies for any non-local build-runner host (and unavailability in case of single system outage).

lgolubenkobit commented 4 months ago

What causes it you should ask the CISA.

In any case the NVD apiKey is for access to NIST NVD CVE-API and has no effect or usage with the CISA KEV json downloads (they are different resources operated by different organisations)

Regarding the ways in which you could have multiple .m2 caches: that would fully depend on how your AWS cloud build pipelines environment is operated.

E.g. I can imagine setups with caches to be datacenter-, serverrack- or even buildrunner-local rather than a global share that is replicated introducing synchronization latencies or a globally shared single share introducing continuous I/O latencies for any non-local build-runner host (and unavailability in case of single system outage).

Thanks for the clarification. I thought the same apiKey was used for both downloads. At cache level we didn't found any error.

Now we are disabling cisa kve download by pom propertie and later we are going to mirror it locally.

gpt8763 commented 3 weeks ago

use this method can solve problem vi dependency-check.sh exec "$JAVACMD" $JAVA_OPTS $DEBUG \ -classpath "$CLASSPATH" \ -Dapp.name="dependency-check" \ -Dapp.pid="$$" \ -Dapp.repo="$REPO" \ -Dapp.home="$BASEDIR" \ -Dbasedir="$BASEDIR" \ -Dkev.url=file:///known_exploited_vulnerabilities.json \ org.owasp.dependencycheck.App \ "$@"

download a file known_exploited_vulnerabilities.json on chrome https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json

alan-czajkowski commented 3 weeks ago

@gpt8763 you have to be careful, because you have to regularly update (re-download) that file in order for it to be effective