Add Vertical Pod Autoscaler support

[mateimicu]

Description

Fixes #162 .

Added the configuration options for Vertical Pod Autoscaler. It will monitor only the main container, but I also included options for any side card that is defined in the chart.

The default configuration will offer recommendations only, as proper scaling depends on the workload.

TODOs

Read the Gruntwork contribution guidelines.

• [x] Update the docs.
• [x] Run the relevant tests successfully, including pre-commit checks.
• [x] Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
• [x] Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added Vertical Pod Autoscaling support to the k8s-service chart.

Migration Guide

-

[mateimicu]

One other thought @mateimicu, and I don't have a strong preference on this, what are your thoughts for HPA and VPA to be conditionally mutually exclusive? Should HPA only be rendered if VPA is false and VPA only rendered if HPA is false? Just a thought.

This is a good point; multiple docs mention that HPA and VPA running concurrently is hard or not recommended. According to VPA docs :

Vertical Pod Autoscaler should not be used with the Horizontal Pod Autoscaler (HPA) on CPU or memory at this moment. However, you can use VPA with HPA on custom and external metrics.

The current HPA and implementation works only on CPU / Memory and not on custom metrics. In this sense, it is almost mutually exclusive with VPA (which also works on CPU / Memory).

I oped to have the VPA work with HPA for the updateMode: Off case, which only recommends the correct size that can be of value regardless of the scaling behavior. This is also the default chart setting.