pinterest / teletraan

Teletraan is Pinterest's deploy system.
Apache License 2.0
1.81k stars 250 forks source link

Improve job pool configurations #1746

Closed tylerwowen closed 3 weeks ago

tylerwowen commented 3 weeks ago

Due to misconfiguration, the job pool used in Teletraan has only one thread. Similar issue fixed in https://github.com/pinternal/rodimus/pull/110

  1. ExecutorService used for thread pool has only 1 thread. Changed the work queue type to fix. The configuration is the same as the regular pool in the Rodimus.
  2. ExecutorService instance is not managed, so it's not properly shutdown. Use Dropwizard for lifecycle management.
  3. Bind it to Micrometer's registry so we get metrics.

Considerations

I didn't have the whole change in Rodimus ported because it was not needed. There are only 3 types of tasks involved in Teletraan, ChangeFeed, NotifyJob and Webhook. They will work fine with just the regular pool configuration.

Test plan

This change is a subset of changes we made to Rodimus and it's been working well. For sanity tests, I deployed a build with this change and verified that change feed events are published and metrics are emitted.

Screenshot 2024-11-08 at 14 39 52 Screenshot 2024-11-08 at 14 39 43