Auto scale based on replicas annotation

[tamalsaha]

Add a new annotation to disable auto scaling. This will make sure that HPA user can autoscale. https://github.com/appscode/voyager/blob/master/docs/guides/ingress/scaling.md#horizontal-pod-autoscaling

[binnythomas-1989]

@tamalsaha Along the same context, a question wether there is an option to add Resource limits for the exporter container so as to make hpa work.

Currently added below to the ingress, but that adds to haproxy and not the exporter container hence fails with the error below

  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for memory
Events:

Current configuration added on the ingress template.

spec:
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 1
      memory: 512Mi

adding the resource limits on the specs worked but that doesn't solve the whole problem, it requires to be added on the exporter container too. Is there a way we could do that? if not added we still get the error.

Please find the template for hpa below

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
  name: abc-voyager
  namespace: abc

spec:
  maxReplicas: 6
  minReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: voyager-abc-appscode
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70