Spike in Deployment Runtime

mikekimlovelytics commented 3 months ago

Sensitive Content Alert Please remember that this repository is publicly visible so do not post anything sensitive in your report such as user emails, secrets, etc.

Overwatch Version v0.8.1.2

Describe the bug Upgrading from v0.8.0.0 to the latest v0.8.1.2 (overwatch_2.12-0.8.1.2.jar ) has steadily spiked our weekly Overwatch deployment time (from 10-12 hrs to now 22 hrs). Requesting to know if our pipeline is as optimized as possible.

Please note the following:

-The pipeline runs every Friday midnight EST. -External optimizer job runs every Thursday midnight EST.

Upgrade was done between Jul 04 and Jul 11 (screenshot1). Please note the Launched "By Scheduler" times for progressive runtime increases.
Bronze cluster setting (screenshot2) and Silver/Gold cluster setting (screenshot3) are included.
All clusters have the following Spark config:
```
overwatch.tableLockTimeout 7200000
spark.driver.maxResultSize 32G
spark.rpc.message.maxSize 1024
```
-Pipeline runs on the following settings:
```
primordial_date: 2024-01-01
max_days: 15
```
Please advise the most cost efficient changes, if any.

Screenshots

Issue Reports Please provide the reports requested from the Troubleshooting page in the docs.

mohanbaabu1996 commented 3 months ago

Hi, @mikekimlovelytics ,

We've noticed a performance issue with the silver job run and silver job status modules. It will be fixed in the next release (0820), which we are hoping to release by the end of next weekend.

mikekimlovelytics commented 3 months ago

Hi, @mikekimlovelytics ,

We've noticed a performance issue with the silver job run and silver job status modules. It will be fixed in the next release (0820), which we are hoping to release by the end of next weekend.

@mohanbaabu1996 Awesome. Do you recommend any changes in our settings?

mohanbaabu1996 commented 3 months ago

I will let you know if any changes are required for the new release. Please keep track of the documentation regarding the new version release.

mohanbaabu1996 commented 1 month ago

Hi @mikekimlovelytics ,

The new version has been released and that will fix the existing issues. Please swap the JAR to 0820 library. https://databrickslabs.github.io/overwatch/changelog/

databrickslabs / overwatch

Spike in Deployment Runtime #1279