search

DependencyTrack / dependency-track

Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain.
https://dependencytrack.org/
Apache License 2.0
2.59k stars 543 forks source link

Can't upload BOM from Python requests #3519

Closed alexvorndran closed 6 months ago

alexvorndran commented 6 months ago

Current Behavior

I wanted to get to know Dependency Track a bit and put up an evaluation instance based on the quickstart. It was easy to get a first working upload based on the API examples and the Swagger doc/Swagger UI, including a cURL command to try from the command line, which also worked great. Next, I wanted to try a more complex BOM for which I wrote a small Python script to encode the payload as described in the docs under Usage > Continuous Integration & Delivery > Large Payloads. Wrote it to disk, passed it to cURL, worked as expected. Then I tried to get rid of cURL and use the requests library to upload the BOM from the Python script directly (see below). Much to my surprise, I got an error 500 back (tried with Python 3.11.8/requests 2.31.0 and Python 3.6.13/requests 2.27.1).

Scripts and Co. The script: ```python #!/usr/bin/env python3 import base64 import json import sys import requests write_payload = True url = "http://localhost:8081/api/v1/sbom" api_key = "odt_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" # replace with real API key project = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" # replace with real project UUID headers = { "Content-Type": "application/json", "X-Api-Key": api_key, "accept": "application/json", } with open(sys.argv[1], "rb") as file: sbom_base64 = base64.b64encode(file.read()).decode("utf-8") payload = {"project": project, "bom": sbom_base64} if write_payload: with open("payload.json", "w") as file: json.dump(payload, file) response = requests.put(url, headers=headers, json=payload) print(response.status_code) print(response.text) ``` The original cURL command: ```text curl -X PUT "http://localhost:8081/api/v1/bom" -H "accept: application/json" -H "X-Api-Key: odt_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" -H "Content-Type: application/json" -d @payload.json; echo ```

The log of the API container showed the following (I tried LOGGING_LEVEL=DEBUG and LOGGING_LEVEL=TRACE):

Log output for failed request ```text dependency-track-dtrack-apiserver-1 | 2024-03-02 09:57:57,781 DEBUG [HttpClientPool] Stats: [leased: 0; pending: 0; available: 0; max: 200] dependency-track-dtrack-apiserver-1 | 2024-03-02 09:57:57,790 ERROR [GlobalExceptionHandler] Uncaught internal server error dependency-track-dtrack-apiserver-1 | javax.ws.rs.NotFoundException: HTTP 404 Not Found dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) dependency-track-dtrack-apiserver-1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1419) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) dependency-track-dtrack-apiserver-1 | at alpine.server.filters.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:225) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) dependency-track-dtrack-apiserver-1 | at alpine.server.filters.ClickjackingFilter.doFilter(ClickjackingFilter.java:93) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) dependency-track-dtrack-apiserver-1 | at alpine.server.filters.WhitelistUrlFilter.doFilter(WhitelistUrlFilter.java:166) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.Server.handle(Server.java:563) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) dependency-track-dtrack-apiserver-1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) dependency-track-dtrack-apiserver-1 | at java.base/java.lang.Thread.run(Unknown Source) dependency-track-dtrack-apiserver-1 | 2024-03-02 09:58:02,781 DEBUG [HttpClientPool] Stats: [leased: 0; pending: 0; available: 0; max: 200] dependency-track-dtrack-apiserver-1 | 2024-03-02 09:58:03,705 DEBUG [HealthServlet] Calling health check: alpine.server.health.checks.DatabaseHealthCheck dependency-track-dtrack-apiserver-1 | 2024-03-02 09:58:03,706 DEBUG [DatabaseHealthCheck] Checking non-transactional connection pool dependency-track-dtrack-apiserver-1 | 2024-03-02 09:58:03,706 DEBUG [DatabaseHealthCheck] Checking transactional connection pool ``` Increasing the debug level from the default didn't yield more information on the error (output shows `LOGGING_LEVEL=TRACE`), when in contrast the feedback on the successful request "blew up" and provided an enormous amount of detail. I can provide the output as well if that helps.

https://github.com/DependencyTrack/dependency-track/issues/2844 might be related, but since the OP didn't provide a detailed log, I'm not sure here.

Steps to Reproduce

  1. Download a sample CycloneDX BOM, e.g. https://github.com/CycloneDX/bom-examples/raw/master/SBOM/cern-lhc-vdm-editor-e564943/bom.xml
  2. Follow the steps to deploy Dependency Track via docker compose as per https://docs.dependencytrack.org/getting-started/deploy-docker/#quickstart-docker-compose
  3. Set a password
  4. Create a project
  5. Obtain the project UUID and API key
  6. Put them in the script above, save it as upload-sbom.py, make the script executable
  7. ./upload-sbom.py bom.xml
  8. 500 and `Uncaught internal server error are shown on the terminal

I also tried sending redacted versions of the request to echo servers (https://httpbin.org/put, https://echo.free.beeceptor.com/) and fiddled with the headers sent by requests until they where identical with the ones sent by cURL, but to no avail.

Expected Behavior

Either it should work or provide a more actionable error message.

Dependency-Track Version

4.10.1

Dependency-Track Distribution

Container Image

Database Server

H2

Database Server Version

No response

Browser

N/A

Checklist

valentijnscholten commented 6 months ago

url = "http://localhost:8081/api/v1/sbom"

What happens if you use the same URL as in the curl command, i.e. http://localhost:8081/api/v1/bom

alexvorndran commented 6 months ago

@valentijnscholten: You are my hero for today! Seems like I got lost in the details and forgot to check the obvious. But you could have put me under oath and I would have sworn you that I copied the route from the cURL command :sweat_smile:

In retrospective, the last comment in https://github.com/DependencyTrack/dependency-track/issues/2844 I linked to above already mentioned a typo in the URL as source of the error. I even thought about it myself because of the 404 in the stacktrace but was to blind to see my own error. My sincere apology for that!

With that in mind, I think I can now relate to some of the points mentioned in https://github.com/DependencyTrack/dependency-track/issues/2604.

github-actions[bot] commented 5 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.