How to avoid having multiple SparkApplication specification YAML files?

[RouhallahAfsharpour]

Hi, I was wondering if there is a way to avoid having multiple SparkApplication specification YAML files.

So for each environment like staging or prod I am using a separate YAML file but most of the values for specifications in both environments are the same.

Is there a way to just have one single YAML file for SparkApplication specification and somehow define different values based on the env?

For example in staging:

  executor:
    cores: 3
    instances: 2
    memory: "16384m"

But in prod:

  executor:
    cores: 3
    instances: 2
    memory: "32768m"

I want to somehow define both in one file like this for example:

  executor:
    cores: 3
    instances: 2
    memory: 
      staging: "16384m"
      prod: "32768m"

Or any other solution that you can think of. But I need to know how spark will understand it ... I would really appreciate if you could help.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.