search

CycloneDX / cdxgen

Creates CycloneDX Bill of Materials (BOM) for your projects from source and container images. Supports many languages and package managers. Integrate in your CI/CD pipeline with automatic submission to Dependency Track server.
https://cyclonedx.github.io/cdxgen/
Apache License 2.0
546 stars 158 forks source link

Unable to extract and archiving Read Error #1314

Open arkajnag23 opened 1 month ago

arkajnag23 commented 1 month ago

Discussed in https://github.com/CycloneDX/cdxgen/discussions/1313

Originally posted by **arkajnag23** August 14, 2024 I am running CDXGEN in Server mode and using API to generate the SBOMs. For some multi-module maven projects which contains Java, JS packages, we are seeing errors , which eventually I feel, preventing to generate SBOM completely. `Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete! Unable to extract /tmp/jar-deps-q95rby/webapp.war. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) Unable to extract /tmp/jar-deps-q95rby/cde-3.3.3.jar. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) Unable to extract /tmp/jar-deps-q95rby/cde-3.2.1.jar. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) Unable to extract /tmp/jar-deps-q95rby/abc-1.2.3.jar. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) ` These errors are not providing a lot information. Require help.
arkajnag23 commented 1 month ago

Also, encountered below error stack trace: Unable to extract /tmp/jar-deps-pWNKTT/731100000-jar-with-dependencies.jar. Skipping. Error: Malicious entry: / at ZipEntry.validateName (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:898:19) at FsRead.readEntriesCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:349:27) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at FSReqCallback.wrapper [as oncomplete] (node:fs:672:5)

prabhu commented 1 month ago

Another valid error. Can you share this problematic jar?

arkajnag23 commented 4 weeks ago

@prabhu Even before trying to fetch the files from temp directories, the files are deleted. Do we have any retention period till how long these files should be available?

arkajnag23 commented 4 weeks ago

Also, there are errors like: Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete!

No further information is provided. Where can we see the error logs?

arkajnag23 commented 4 weeks ago

Another valid error. Can you share this problematic jar?

@prabhu As the jar is internal, it wouldn't be possible to share. What is expected to be verified or debugged here?

prabhu commented 4 weeks ago

Are you mounting the /tmp directory to the container image using -v /tmp:/tmp? The files must be getting corrupted either during download or during the copy and extract operation. Best to work with a devops person to troubleshoot this further since this is not a cdxgen issue.

arkajnag23 commented 4 weeks ago

@prabhu Let me mount /tmp directory. Btw, when you say extract, are we decompiling the jar?

arkajnag23 commented 4 weeks ago

Also, there are errors like: Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete!

No further information is provided. Where can we see the error logs?

@prabhu Do we generate any error logs if not console log out, as the error mentions multiple errors happened, but what are the errors , how can we find them?

arkajnag23 commented 4 weeks ago

/tmp

@prabhu Found out for this one, it is a corrupted jar file. But we are seeing a lot of Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete!

And in our CDXGEN SERVER mode API, call we not using requiredOnly or babel parameter. Falling back to parsing pom.xml files. Only direct dependencies would get included! Executing '/opt/maven/3.9.8/bin/mvn -fn org.cyclonedx:cyclonedx-maven-plugin:2.8.0:makeAggregateBom -DoutputName=bom -q -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-ui-apidoc Executing 'mvn dependency:tree -DoutputFile=/tmp/cdxmvn-dNIoGd/mvn-tree.txt -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-ui-apidoc

prabhu commented 4 weeks ago

Can you run in cli mode with the environment variable CDXGEN_DEBUG_MODE=debug. Usually these might be related to the version of java needed for the given application.

arkajnag23 commented 4 weeks ago

Can you run in cli mode with the environment variable CDXGEN_DEBUG_MODE=debug. Usually these might be related to the version of java needed for the given application.

@prabhu Actually for our requirement, we can't use the CLI mode, we are using docker run server mode, and I launched container in bash mode, and ran export CDXGEN_DEBUG_MODE=debug;export PREFER_MAVEN_DEPS_TREE=true;

Strangely the console log is saying that its generating maven dependency tree, but when I check the final aggregated SBOM, it doesn't include the information.

$ curl -X POST http://localhost:9090/sbom -H "Content-Type: application/json" -d '{
  "path": "/var/EventHub/event-hub-core/event-hub-core/",
  "type": "maven,js",
  "multiProject": true,
  "resolveTransitive": true,
  "recurse": true
}' 
Falling back to parsing pom.xml files. Only direct dependencies would get included!
Executing '/opt/maven/3.9.8/bin/mvn -fn org.cyclonedx:cyclonedx-maven-plugin:2.8.0:makeAggregateBom -DoutputName=bom -q -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-clients
Executing 'mvn dependency:tree -DoutputFile=/tmp/cdxmvn-cCcF7m/mvn-tree.txt -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-clients

I ran local mvn dependency:tree as shared above but below dependencies and transitive are not included.

--- maven-dependency-plugin:2.10:tree (default-cli) @ event-hub-clients ---
[INFO] event-hub:event-hub-clients:jar:4.0-SNAPSHOT
[INFO] +- org.apache.kafka:kafka-clients:jar:3.4.0.2:compile
[INFO] |  +- com.github.luben:zstd-jni:jar:1.5.2-1:runtime
[INFO] |  \- org.lz4:lz4-java:jar:1.8.0:runtime
[INFO] +- org.apache.zookeeper:zookeeper:jar:3.6.3:compile
[INFO] |  +- org.apache.zookeeper:zookeeper-jute:jar:3.6.3:compile
[INFO] |  \- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] +- org.apache.curator:curator-recipes:jar:5.2.1:compile
[INFO] |  \- org.apache.curator:curator-framework:jar:5.2.1:compile
[INFO] |     \- org.apache.curator:curator-client:jar:5.2.1:compile
prabhu commented 4 weeks ago

Can you triage further? Looks like you are making good progress. It is difficult to support without access to the source code.

arkajnag23 commented 4 weeks ago

triage

@prabhu Unfortunately can't share the source code. :( Kind of deadlock situation, logs are not sharing too much information and also the temporary txt file are removed right now, making it more difficult to debug.

prabhu commented 4 weeks ago

What is resolveTransitive: true? It is not part of cdxgen. Check the base directory since that influences the dependency tree generated by maven. Also try the old cyclonedx maven plugin approach to see if it works better for this app.

arkajnag23 commented 4 weeks ago

@prabhu I saw some where that resolveTransitives is for transitive dependencies, it's my mistake. I was using cyclonedx maven plugin before and it was good, but went with cdxgen as it supports multiple project types and our project is running multiple types with maven.

prabhu commented 4 weeks ago

Can you unset PREFER_MAVEN_DEPS_TREE environment variable? cdxgen will then use the cyclonedx maven plugin automatically.

prabhu commented 3 weeks ago

@arkajnag23 could you kindly retest with the latest 10.9.5. Wondering if this was the fix needed to improve this.

arkajnag23 commented 3 weeks ago

Thanks @prabhu will do and let you know.