charts/portal - Add tolerations to each statefulset and deployment

[hydak]

Description of the change

I added the ability for portal helm charts to have tolerations

Benefits

Tolerations allow pods to schedule onto nodes with matching taints

Drawbacks

None

Applicable issues

None

Additional information

Checklist

• [X] Variables are documented in the README.md
• [X] Title of the PR starts with chart name (e.g. [charts/gateway])
• [X] If the chart contains a values-production.yaml apart from values.yaml, ensure that you implement the changes in both files

[scottchc1]

for BOTH nodeSeletor and tolerations, do we ever expect different values per component? cleaner to just have one entry in global section (instead of pasting the same custom values 30 times).

you probably still need both nodeSelector and tolerations in global values to pass to subcharts (for ingress controller, mysql, rabbitmq), or add each entry to those subchart's values section

updated druid subchart needs to be tar'ed into charts subdir

[hydak]

@scottchc1 we are implementing the standard for tolerations. In my research, I do see some people like to have certain node types for certain types of containers. We don't have an official stance on that, so we're just making it the most configurable thing we can to allow for flexibility.

According to @Gazza7205 we don't need to repackage that subchart as the portal chart.yaml references the druid charts via a relative path ../druid. The druid subcharts you are referring to were included here in case customers wanted to use their own Apache Druid deployment oran official Druid chart.

[scottchc1]

re: most configurable tolerations - ok. probably still nice to list global.tolerations and global.nodeSelector for subcharts, even though commented