add prometheus annotations to Ingress controllers

[OliverMKing]

Description

This allows users to easily consume web app routing nginx-ingress prometheus metrics. It follows the guidance in the nginx docs.

E2e tests prove a user-created prometheus instance can connect to the web app routing nginx-ingress controller metrics. There are no failure scenarios to test. The failure scenario is a user-created prometheus instance is unable to connect to the ingress controller metric which will be caught by this test if it is true. A misconfigured prometheus instance would be a customer issue, not a problem with web app routing. There are no failure cases that web app routing can handle itself because this functionality is completely dependent on customer prometheus.

You can see local e2e tests passing.

It won't disrupt or change the experience for users not using Prometheus in any way.

Type of change

Please delete options that are not relevant.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [x] This change requires a documentation update

We should document this feature on our next release with examples.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration. Is it a breaking change which will impact consuming tool(s)?

• [x] Tested locally
• [x] Unit tested
• [x] E2e tested

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules

[OliverMKing]

Please add to the description of the PR how the new prom test integrate with existing tests and the failure scenarios covered. It would also be helpful, if possible, to see successful E2E runs from a local environment. Otherwise, nice PR!

Updated the description