Spike in Deployment Runtime

[mikekimlovelytics]

Sensitive Content Alert Please remember that this repository is publicly visible so do not post anything sensitive in your report such as user emails, secrets, etc.

Overwatch Version v0.8.1.2

Describe the bug Upgrading from v0.8.0.0 to the latest v0.8.1.2 (overwatch_2.12-0.8.1.2.jar ) has steadily spiked our weekly Overwatch deployment time (from 10-12 hrs to now 22 hrs). Requesting to know if our pipeline is as optimized as possible.

Please note the following:

-The pipeline runs every Friday midnight EST. -External optimizer job runs every Thursday midnight EST.

• Upgrade was done between Jul 04 and Jul 11 (screenshot1). Please note the Launched "By Scheduler" times for progressive runtime increases.
• Bronze cluster setting (screenshot2) and Silver/Gold cluster setting (screenshot3) are included.
• All clusters have the following Spark config:

  overwatch.tableLockTimeout 7200000
  spark.driver.maxResultSize 32G
  spark.rpc.message.maxSize 1024

  -Pipeline runs on the following settings:

  primordial_date: 2024-01-01
  max_days: 15

  Please advise the most cost efficient changes, if any.

Screenshots

Issue Reports Please provide the reports requested from the Troubleshooting page in the docs.

[mohanbaabu1996]

Hi, @mikekimlovelytics ,

We've noticed a performance issue with the silver job run and silver job status modules. It will be fixed in the next release (0820), which we are hoping to release by the end of next weekend.

[mikekimlovelytics]

Hi, @mikekimlovelytics ,

We've noticed a performance issue with the silver job run and silver job status modules. It will be fixed in the next release (0820), which we are hoping to release by the end of next weekend.

@mohanbaabu1996 Awesome. Do you recommend any changes in our settings?

[mohanbaabu1996]

I will let you know if any changes are required for the new release. Please keep track of the documentation regarding the new version release.