OWASP-Benchmark / BenchmarkJava

OWASP Benchmark is a test suite designed to verify the speed and accuracy of software vulnerability detection tools. A fully runnable web app written in Java, it supports analysis by Static (SAST), Dynamic (DAST), and Runtime (IAST) tools that support Java. The idea is that since it is fully runnable and all the vulnerabilities are actually exploitable, it’s a fair test for any kind of vulnerability detection tool. For more details on this project, please see the OWASP Benchmark Project home page.
https://owasp.org/www-project-benchmark/
GNU General Public License v2.0
660 stars 1.05k forks source link

LDAP Test Cases Don't Work on Java 10+ #54

Closed davewichers closed 5 months ago

davewichers commented 6 years ago

I'm seeing errors starting the LDAP server on Java 10 on my Mac. I'll have to test on Linux/Windows to see if it's platform specific or not. So, even if Benchmark starts up just fine on Java9/10, the LDAP test cases might all be failing, so tools would score zero in that category when they shouldn't. As a fallback for now, run Benchmark with Java 8, which is the version of Java its been tested most on.

Note: This should only affect DAST and IAST tools, not SAST tools.

davewichers commented 4 years ago

The upgrade will involve using the new 2.0.0 release of Apache Directory Server. See: https://search.maven.org/artifact/org.apache.directory.server/apacheds-core. The dev team is still working on this release, having released their 26th release candidate: 2.0.0.AM26 in Feb 2020. This version is probably good enough to work on the upgrade now, although an actual 2.0.0 release would be preferred.

gitnachogo commented 4 years ago

Hi! I am integrating a new SAST tool and I've ended up in this issue because I have a problem with the LDAP test cases when creating the scorecards. In my case, the tool reports CWE-90 only once, and the amount of TP and FP is 0 (at least one of them should be 1). But I've tried reporting 90 for every issue found, just to test what'd happen if there were more than just 1 issue reported as LDAP; and the result is that the TP is 4 and FP is 1, counting on there are literally 1000+ issues reported as 90 (wrongly, but just to test this). I've tried installing OpenJDK 8 and it's still not working. I know it shouldn't be related, because I'm not running the tool, but I just saw it here and thought this was the right place to do it. Thanks in advance @davewichers ! Hope to be soon contributing with this new tool integrated hehehe

davewichers commented 4 years ago

The only way I can help is for you to send me the results file and I can write the parser for it. Or you submit a pull request with the parser you've written so far and the results file so I can debug the issue. LDAP test cases not working with Java 10+ and being able to score LDAP findings with a new SAST tool are completely independent issues unrelated to each other. Feel free to contact me directly at dave dot wichers at owasp dot org.

dandersonaspect commented 3 years ago

After updating to Java 10+, the error message we were getting with Apacheds 1.5.7 is:

java.lang.NumberFormatException: multiple points

The latest Apache 2.0.0.AM26 does not report this error. However, it reports a different error:

java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

Apacheds is published to Maven Central as apacheds-all, which is a fat JAR created by repackaging the classes from all custom and third-party dependency code (using the maven-shade plugin). The error above is caused because the Bouncy Castle signed JAR fails verification against the signature files. The implemented fix was to exclude the signature files from the fat JAR. This was apparently fixed post 2.0.0.AM25, but unfixed in 2.0.0.AM26 because Bouncy Castle renamed their signature files. It was re-fixed in Oct 2020 in a 2.0.0.AM27-SNAPSHOT.

See: https://issues.apache.org/jira/browse/DIRSERVER-2247 https://issues.apache.org/jira/browse/DIRSERVER-2326

davewichers commented 5 months ago

Fixed with commit 2c67145