It doesn't seem to scan sub projects

[peterlai-roboops]

So as a test, we added a failing library to a sub project's pom in the repo. SL scan doesn't seem pick it up, when we then removed it, and added it to the main pom in the root folder, it picks it up.

[prabhu]

Hi @peterlai-roboops

Are you saying dep-scan is not identifying the sub pom.xml? To troubleshoot this, can you run scan with the environment variable

SCAN_DEBUG_MODE=debug

Then inspect the bom-java.json in reports directory to see if all the packages are identified correctly?

[prabhu]

Hi @peterlai-roboops ,

Did you get a chance to test this? Would you prefer a zoom based support?

[peterlai-roboops]

i'm running this: docker run --rm -e "WORKSPACE=$(pwd)" -e "SCAN_DEBUG_MODE=debug" -v "$(pwd):/app" shiftleft/scan scan -m ci --type depscan Dependency Scan Summary (java) ╔═════════════╤═══════╤════════╗ ║ Severity │ Count │ Status ║ ╟─────────────┼───────┼────────╢ ║ UNSPECIFIED │ 0 │ ✅ ║ ║ LOW │ 0 │ ✅ ║ ║ MEDIUM │ 3 │ ✅ ║ ║ HIGH │ 4 │ ❌ ║ ║ CRITICAL │ 0 │ ✅ ║ ╚═════════════╧═══════╧════════╝ [10:43:48] DEBUG Unable to send telemetry DEBUG Depscan and HTML report written to file: /app/reports/depscan-report-java.json, /app/reports/depscan-report-java.html 👍 DEBUG Unable to send telemetry DEBUG License check and HTML report written to file: /app/reports/license-java.json, /app/reports/license-java.html 👍 [10:43:48] DEBUG Aggregate report written to /app/reports/scan-full-report.json DEBUG No scan summary was produced - [], /app/reports/scan-full-report.json

Its meant to detect logback 1.0.9 but it doesn't, i can't seem to see bom-java.json in the reports folder.

[peterlai-roboops]

These are the scenarios i've tested with: code pre-compiled then

• parent pom updated: not detect
• child pom updated: not detect code un-compile then
• parent pom updated: detect
• child pom updated: not detect code re-compile
• parent pom updated: detect
• child pom updated: not detect

I've only added the dependencies in the pom, and I'm not actually using the dependency in the code. This is an example of when it does detect it (In this instence, code un-compile, parent pom updated, child pom not updated) ![Uploading 2020-10-01 11_51_59-Windows PowerShell.png…]()

[prabhu]

Ah windows. Apologies, Windows is currently broken. Are you able to use WSL2 or any CI such as GitHub?

[prabhu]

Also scan uses openjdk 11. Does your project need jdk 1.8 or a different version?

[peterlai-roboops]

Yea i can put it through our jenkins, thats linux based. Our project needs jdk 1.8, we've also tried running with the --build parameter, that doesn't seem to have an effect.

[prabhu]

Cool. Out of curiosity try replacing shiftleft/scan with shiftleft/scan-java and use --build parameter. Let me know if you get different results. scan-java image contains jdk 1.8 and 11.

[peterlai-roboops]

Running: sh 20:21:33 + DISABLE_TELEMETRY=1 20:21:33 + SCAN_DEBUG_MODE=debug 20:21:33 + cd src 20:21:33 + scan --build -m ci --type credscan,depscan

cat ./src/reports/bom-java.json 20:23:11 { 20:23:11 "bomFormat": "CycloneDX", 20:23:11 "specVersion": "1.2", 20:23:11 "serialNumber": "urn:uuid:7c1161e1-1c63-4869-b4e4-3e3871ccf5c7", 20:23:11 "version": 1, 20:23:11 "metadata": { 20:23:11 "timestamp": "2020-10-04T16:21:37.025Z", 20:23:11 "tools": [ 20:23:11 { 20:23:11 "tool": { 20:23:11 "vendor": "AppThreat", 20:23:11 "name": "cdxgen", 20:23:11 "version": "2.0.13" 20:23:11 } 20:23:11 } 20:23:11 ], 20:23:11 "authors": [ 20:23:11 { 20:23:11 "author": { 20:23:11 "name": { 20:23:11 "name": "Team AppThreat" 20:23:11 }, 20:23:11 "email": "cloud@appthreat.com" 20:23:11 } 20:23:11 } 20:23:11 ] 20:23:11 }, 20:23:11 "components": [ 20:23:11 { 20:23:11 "group": "jfoundation", 20:23:11 "name": "jf-log", 20:23:11 "version": "5.0.0-SNAPSHOT", 20:23:11 "licenses": null, 20:23:11 "purl": "pkg:maven/jfoundation/jf-log@5.0.0-SNAPSHOT?type=jar", 20:23:11 "scope": "optional", 20:23:11 "type": "library", 20:23:11 "bom-ref": "pkg:maven/jfoundation/jf-log@5.0.0-SNAPSHOT?type=jar" 20:23:11 }, 20:23:11 { 20:23:11 "group": "org.apache.cxf", 20:23:11 "name": "cxf-api", 20:23:11 "version": "2.4.9", 20:23:11 "licenses": null, 20:23:11 "purl": "pkg:maven/org.apache.cxf/cxf-api@2.4.9?type=jar", 20:23:11 "scope": "optional", 20:23:11 "type": "library", 20:23:11 "bom-ref": "pkg:maven/org.apache.cxf/cxf-api@2.4.9?type=jar", 20:23:11 "description": "Apache CXF API" 20:23:11 } 20:23:11 ] 20:23:11 }

[peterlai-roboops]

Confirmation that a sub-module is downloading the logback dependecy. This should fail as critical.

--- maven-dependency-plugin:2.8:resolve (default-cli) @ ms-price-interfaces --- 20:14:20 [INFO] ch.qos.logback:logback-classic:jar:1.0.9:compile

[prabhu]

ok, i can see that the bom generation is failing so the tool is falling back to parsing just the root pom.xml.

Can you try using https://github.com/AppThreat/cdxgen to generate a bom file correctly for this project. The generated json file should include logback-classic if it executes correctly. If not we need a zoom session to troubleshoot the issue.

[peterlai-roboops]

cd src 08:58:31 + cdxgen -o bom.xml 08:58:32 Executing 'mvn org.cyclonedx:cyclonedx-maven-plugin:makeAggregateBom' in /home/jenkins/agent/workspace/peter_dxb_feature_add_slscan/src 08:58:34 BOM file written to bom.xml 08:58:34 + cat bom.xml 08:58:34 <?xml version="1.0" encoding="utf-8"?> 08:58:34 08:58:34 08:58:34 08:58:34 . 08:58:34 Base path 08:58:34 08:58:34 08:58:34 pom.xml 08:58:34 Package file 08:58:34 08:58:34 08:58:34 08:58:34 2020-10-05T04:58:34.372Z 08:58:34 08:58:34 08:58:34 AppThreat 08:58:34 cdxgen 08:58:34 2.0.14 08:58:34 08:58:34 08:58:34 08:58:34 08:58:34 08:58:34 Team AppThreat 08:58:34 08:58:34 email>cloud@appthreat.com</email 08:58:34 08:58:34 08:58:34 08:58:34 08:58:34 08:58:34 jfoundation 08:58:34 jf-log 08:58:34 5.0.0-SNAPSHOT 08:58:34 purl>pkg:maven/jfoundation/jf-log@5.0.0-SNAPSHOT?type=jar</purl 08:58:34 08:58:34 08:58:34 08:58:34 org.apache.cxf 08:58:34 cxf-api 08:58:34 2.4.9 08:58:34 purl>pkg:maven/org.apache.cxf/cxf-api@2.4.9?type=jar</purl 08:58:34 08:58:34 <![CDATA[Apache CXF API]]> 08:58:34 08:58:34 08:58:34 08:58:34

[prabhu]

Can you email me prabhu @ shiftleft .io so that we can have a zoom session to troubleshoot?

[gitnachogo]

Hi! I think I have the same issue. Here you got my test cases:

1. Compiled projects using --type java,depscan: The bom-java.json and .xml are generated, containing the os dependencies in the parent's pom, but nothing else is generated. The dependency extracted is bom-ref": "pkg:maven/com.hierynomus/sshj@0.24.0?type=jar", so probably this one is not vulnerable, and that's why nothing more (like html file) is generated, but many os deps from the submodules are missing. Output below:

2. Compiled projects using a for loop to scan all of them one by one, generating reports in different outputs: The bom-java.json and xml files are generated in every iteration, containing all the os dependencies (5 and 7 respectively), but nothing more (like html file) is generated. Example of one of the iterations is below:

[prabhu]

@gitnachogo How are the sub-modules referred in the main pom.xml? Only in the <module> section or under <dependency> as well?

[prabhu]

Yes, html file is no longer getting created. This is because after we introduced multiprocessing based concurrency the code that collects all depscan json files to convert is running even before depscan finishes. Have to fix this bug.

[gitnachogo]

@gitnachogo How are the sub-modules referred in the main pom.xml? Only in the <module> section or under <dependency> as well?

It's defined in both places

[prabhu]

ok must see the pom.xml. Can you email me so that we can have a zoom session? prabhu at shiftleft.io