Running Spark in multiple pods of K8s - Githubissues

uc-cdis / cloud-automation

Automation for standing up Gen3 commons

Apache License 2.0

41 stars 67 forks source link

Running Spark in multiple pods of K8s #2557

Open thanh-nguyen-dang opened 4 months ago

thanh-nguyen-dang commented 4 months ago

Link to JIRA ticket if there is one: https://ctds-planx.atlassian.net/browse/MIDRC-557 https://ctds-planx.atlassian.net/browse/MIDRC-730

New Features

We run Spark in multiple pods in kubernetes cluster.
By running Spark cluster in multiple pods, we need smaller node. Hence, not required big node anymore. Consequently, it resolve the performance issue of ETL process

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes