[Enhancement][opensearch] Allow overriding affinity definition

[DCCSKrezovic]

Is your feature request related to a problem? Please describe.

It is currently only possible to configure affinity as "soft" or "hard". But it's not possible to add additional affinity rules, i.e to avoid scheduling on nodes with other resource intensive pods

Describe the solution you'd like

To specify my own affinity rules.

[prudhvigodithi]

[Triage] Thanks @DCCSKrezovic for raising an issue, can you please contribute to the chart based on existing chart settings? Thank you @bbarani @rishabh6788

[emmahsax]

Would be very helpful to be able to set our own affinity rules.

[prudhvigodithi]

Hey @emmahsax , @premkirank can you please take a look and see if you can contribute?

[emmahsax]

@prudhvigodithi We are already working on a fork that allows this. We are also interested in allowing passing of custom anti-affinity, AKA not being required to use one of the two hard and soft options provided. We're doing our best to make it a non-breaking change so it could theoretically be merged into upstream.

[prudhvigodithi]

Hey @emmahsax once you have a PR, please raise it against https://github.com/opensearch-project/helm-charts/, I can test your change, review and get it merged. Thanks

[emmahsax]

@prudhvigodithi Here's that PR: https://github.com/opensearch-project/helm-charts/pull/495!