search

DependencyTrack / dependency-track

Dependency-Track is an intelligent Component Analysis platform that allows organizations to identify and reduce risk in the software supply chain.
https://dependencytrack.org/
Apache License 2.0
2.44k stars 530 forks source link

Upgrade DependencyTrack apiserver fails without exception on computing severity where value is NULL #3821

Closed ThEcTecan closed 3 weeks ago

ThEcTecan commented 4 weeks ago

Current Behavior

Upgrading Dependency Track apiserver from version 4.10.1 to 4.11.x fails with no logged exception. You can see that the v4110Updater has started it's execution, but the logs stop with

Computing severities for vulnerabilities where severity is currently NULL.

Afterwards the docker container is shutdown.

Here are the full logs of the upgrade process:

2024-06-03 10:23:15,986 [] INFO [alpine.embedded.EmbeddedJettyServer] alpine-executable-war v2.2.5 (c3a1a709-acdc-4ca7-96dd-5eaab858ee32) built on: 2024-02-29T20:30:01Z 2024-06-03 10:23:18,188 [] INFO [alpine.Config] -------------------------------------------------------------------------------- 2024-06-03 10:23:18,190 [] INFO [alpine.Config] OS Name: Linux 2024-06-03 10:23:18,193 [] INFO [alpine.Config] OS Version: 5.15.153.1-1.cm2 2024-06-03 10:23:18,195 [] INFO [alpine.Config] OS Arch: amd64 2024-06-03 10:23:18,196 [] INFO [alpine.Config] CPU Cores: 2 2024-06-03 10:23:18,205 [] INFO [alpine.Config] Max Memory: 5.1 GB (5,480,906,752.0 bytes) 2024-06-03 10:23:18,207 [] INFO [alpine.Config] Java Vendor: Eclipse Adoptium 2024-06-03 10:23:18,209 [] INFO [alpine.Config] Java Version: 21.0.3+9-LTS 2024-06-03 10:23:18,213 [] INFO [alpine.Config] Java Home: /opt/java/openjdk 2024-06-03 10:23:18,215 [] INFO [alpine.Config] Java Temp: /tmp 2024-06-03 10:23:18,216 [] INFO [alpine.Config] User: dtrack 2024-06-03 10:23:18,218 [] INFO [alpine.Config] User Home: /data/ 2024-06-03 10:23:18,219 [] INFO [alpine.Config] -------------------------------------------------------------------------------- 2024-06-03 10:23:18,220 [] INFO [alpine.Config] Initializing Configuration 2024-06-03 10:23:18,222 [] INFO [alpine.Config] System property alpine.application.properties not specified 2024-06-03 10:23:18,223 [] INFO [alpine.Config] Loading application.properties from classpath 2024-06-03 10:23:18,233 [] INFO [alpine.Config] -------------------------------------------------------------------------------- 2024-06-03 10:23:18,235 [] INFO [alpine.Config] Application: Dependency-Track 2024-06-03 10:23:18,236 [] INFO [alpine.Config] Version: 4.11.3 2024-06-03 10:23:18,237 [] INFO [alpine.Config] Built-on: 2024-06-03T09:33:35Z 2024-06-03 10:23:18,238 [] INFO [alpine.Config] -------------------------------------------------------------------------------- 2024-06-03 10:23:18,240 [] INFO [alpine.Config] Framework: Alpine 2024-06-03 10:23:18,241 [] INFO [alpine.Config] Version : 2.2.5 2024-06-03 10:23:18,243 [] INFO [alpine.Config] Built-on: 2024-02-29T20:30:01Z 2024-06-03 10:23:18,244 [] INFO [alpine.Config] -------------------------------------------------------------------------------- 2024-06-03 10:23:18,370 [] WARN [io.micrometer.core.instrument.binder.jvm.ExecutorServiceMetrics] Failed to bind as java.util.concurrent.Executors$AutoShutdownDelegatedExecutorService is unsupported. 2024-06-03 10:23:18,416 [] INFO [org.dependencytrack.RequirementsVerifier] Initializing requirements verifier 2024-06-03 10:23:18,418 [] INFO [alpine.server.metrics.MetricsInitializer] Registering system metrics 2024-06-03 10:23:18,491 [] INFO [org.dependencytrack.upgrade.UpgradeInitializer] Initializing upgrade framework 2024-06-03 10:23:28,058 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v400.v400Updater does not need to run. 2024-06-03 10:23:28,064 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v410.v410Updater does not need to run. 2024-06-03 10:23:28,086 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v420.v420Updater does not need to run. 2024-06-03 10:23:28,114 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v440.v440Updater does not need to run. 2024-06-03 10:23:28,121 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v450.v450Updater does not need to run. 2024-06-03 10:23:28,133 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v460.v460Updater does not need to run. 2024-06-03 10:23:28,138 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v463.v463Updater does not need to run. 2024-06-03 10:23:28,142 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v470.v470Updater does not need to run. 2024-06-03 10:23:28,144 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v480.v480Updater does not need to run. 2024-06-03 10:23:28,147 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v490.v490Updater does not need to run. 2024-06-03 10:23:28,150 [] DEBUG [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v4100.v4100Updater does not need to run. 2024-06-03 10:23:28,187 [] INFO [alpine.server.upgrade.UpgradeExecutor] Upgrade class org.dependencytrack.upgrade.v4110.v4110Updater about to run. 2024-06-03 10:23:28,187 [] INFO [org.dependencytrack.upgrade.v4110.v4110Updater] Dropping foreign key constraint from "VULNERABILITY"."CWE" 2024-06-03 10:23:28,191 [] INFO [org.dependencytrack.upgrade.v4110.v4110Updater] Dropping index "VULNERABILITY"."VULNERABILITY_CWE_IDX" 2024-06-03 10:23:28,193 [] INFO [org.dependencytrack.upgrade.v4110.v4110Updater] Dropping column "VULNERABILITY"."CWE" 2024-06-03 10:23:28,196 [] INFO [org.dependencytrack.upgrade.v4110.v4110Updater] Dropping table "CWE" 2024-06-03 10:23:28,222 [] INFO [org.dependencytrack.upgrade.v4110.v4110Updater] Computing severities for vulnerabilities where severity is currently NULL

``

Steps to Reproduce

  1. Start the upgrade from dependencytrack apiserver 4.10.1 to 4.11.3.

Expected Behavior

The upgrade succeeds or there is an error message logged, indicating why the upgrade fails.

Dependency-Track Version

4.11.3

Dependency-Track Distribution

Container Image

Database Server

Microsoft SQL Server

Database Server Version

Azure SQL Database (latest)

Browser

Microsoft Edge

Checklist

nscuro commented 4 weeks ago

When the upgrade fails, it will log a stack trace, and it will not automatically shut down.

Are you running on an orchestrator such as Kubernetes, that could be killing the pod if it doesn't become healthy in a certain amount of time?

How much time passes between Computing severities for vulnerabilities where severity is currently NULL being logged and the container being shut down?

ThEcTecan commented 3 weeks ago

Are you running on an orchestrator such as Kubernetes, that could be killing the pod if it doesn't become healthy in a certain amount of time?

Thanks, that is exactly what happened.

I am indeed using docker compose to start the container in an Azure App Service. According to Azure App Service on Linux FAQ there is a default timeout of 230s for the container to start up.

After increasing the timeout, I could see that Computing severities for vulnerabilities where severity is currently NULL took ~12 minutes and the upgrade ran through successful.

nscuro commented 3 weeks ago

Hmmm, 12min is a lot for this operation. It already utilizes SQL statement batching so it doesn't perform a lot of requests to the database.

Do you have high latency between API server and database?