Adjust APS MONIT metrics streaming

[vkuznet]

Based on Nikodemas report:

We have created a dashboard with the new APS dashboard (original dashboard is here). A few things about the fields in the new dashboard/logs with the examples:

• missing data.cmsweb_env - k8s-preprod value in the original dashboard, no alternatives in the new dashboard
• missing data.cmsweb_cluster - cmsweb-testbed value in the original dashboard, but data.x_forwarded_host could work as it currently has the value of cmsweb-test10.cern.ch in the new dashboard
• missing data.client_version - v002 value in the original dashboard, but we could use data.user_agent in the new dashboard that has a value of WMCore.Services.Requests/v002
• missing data.proxy_server - x509-proxy-server or aps-proxy-server, no alternatives in the new dashboard

We need to implement those to be compatible with APS+filebeat+logstash approach used in APS timber dashboard.

[vkuznet]

Added in this commit: 791702a202e4c9986bb4d8f10757a93a837e3ec4

[vkuznet]

@arooshap , @nikodemas : I added missing fields for sending APS logs directly to MONIT in this ticket. The cmsweb_env, cmsweb_cluster, and proxy_server will be set if we setup CMSWEB_ENV, CMSWEB_CLUSTER, and APS_SERVER in aps k8s yaml files. And, client_version will be set via user-agent of HTTP request.

Once this issue is committed, APS go.mod and go.sum will need an appropriate update and new APS image should be built (I'll do it later). After that you may test new feature in cmsweb clusters and decide should it be used. For that a set of stress tests will be required to determine the latency of APS+MONIT vs APS+filebeat+logstash+MONIT routes.