Custom metric creation is not working

[mmavrudiev]

Hi all,

I have tried to create a custom-metric using the below procedure but I can't see any value in the database to be populated, can you advise what could be the issue?

Autoscaler version v5.0.3

1. Create a policy: ❯ cat custom_metric.json { "instance_min_count":1, "instance_max_count":4, "scaling_rules":[ { "metric_type":"testmetric", "breach_duration_secs":60, "threshold":100, "operator":">=", "cool_down_secs":120, "adjustment":"+1" } ] }

2. Bind the autoscaler service to an app with the metric policy cf bind-service cf-nodejs autoscaler-test -c custom_metric.json

Can see that some records are already created in the db: autoscaler=> select * from app_metric where metric_type='testmetric_2'; app_id | metric_type | unit | timestamp | value --------------------------------------+--------------+------+---------------------+------- a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001806553860452 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001846542041787 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001886538669255 | (3 rows)

3. Trying to push some metric curl -u username:password "https://autoscalermetrics.system.domain/v1/apps/a5632a69-e90f-48bd-ba07-45eb3d5a8786/metrics" -X POST -d @metric.json -H "Content-Type: application/json"

metric.json: ❯ cat metric.json { "instance_index": 0, "metrics": [ { "name": "testmetric_2", "value": 333, "unit": "jobs" } ] }

getting

• upload completely sent off: 119 out of 119 bytes < HTTP/1.1 200 OK

If I change the user/pass I am getting Unauthorized so seems the above one is correct. The instance index I have verified from the app container, env var.. The app guid is taken directly from the cf..

But in the database there are no values at all: autoscaler=> select * from app_metric where metric_type='testmetric_2'; app_id | metric_type | unit | timestamp | value --------------------------------------+--------------+------+---------------------+------- a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001806553860452 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001846542041787 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001886538669255 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001926536958248 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647001966536982463 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647002006536657510 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647002046540756381 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647002086536247502 | a5632a69-e90f-48bd-ba07-45eb3d5a8786 | testmetric_2 | | 1647002126536876872 | (9 rows)

autoscaler=>

Those records above seems are creating due to some get function, so actually automatially it is trying to get the values from the app? but I want to PUT/POST them.

While debugging I was the following errors in asapi node - loggregator_agent logs: 2022/03/11 13:17:44.042103 failed to connect: unable to lookup a log consumer 2022/03/11 13:17:44.042158 failed to connect: unable to lookup a log consumer

But not sure if that is related to the metrics?

All built-in metrics like cpu, memory in use and etc., are properly displayed, the scaling is properly working based on them.

On the asmetrics node I can't see any errors in the logs...

I am attaching the running manifest of app-autoscaler that we are using in CF as a bosh deployment.

`addons:

• jobs:
  • name: bosh-dns-aliases properties: aliases:
    • domain: autoscalerpostgres.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: postgres_autoscaler network: dynamic query: '*'
    • domain: apiserver.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asapi network: dynamic query: '*'
    • domain: autoscalerscheduler.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asactors network: dynamic query: '*'
    • domain: servicebroker.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asapi network: dynamic query: '*'
    • domain: eventgenerator.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asmetrics network: dynamic query: '*'
    • domain: scalingengine.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asactors network: dynamic query: '*'
    • domain: reverse-log-proxy.service.cf.internal targets:
    • deployment: cf domain: bosh instance_group: log-api network: dynamic query: '*'
    • domain: metricsgateway.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asnozzle network: dynamic query: '*'
    • domain: metricsserver.service.cf.internal targets:
    • deployment: app-autoscaler domain: bosh instance_group: asmetrics network: dynamic query: '*'
    • domain: nats.service.cf.internal targets:
    • deployment: cf domain: bosh instance_group: nats network: static query: '*'
    • domain: _.nats.service.cf.internal targets:
    • deployment: cf domain: bosh instancegroup: nats network: static query: release: bosh-dns-aliases name: bosh-dns-aliases
• jobs:
  • name: bpm release: bpm name: bpm features: use_dns_addresses: true instance_groups:
• azs:
  • z1 instances: 1 jobs:
  • name: postgres properties: databases: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) release: postgres
  • name: shield-agent properties: agent: key: ((/app-autoscaler/SHIELD_agent_key.public_key)) core: ca: ((/app-autoscaler/SHIELD_CA.ca)) env: http_proxy: http://proxy_address:8080 https_proxy: http://proxy_address:8080 name: autoscaler-shield-agent-((/app-autoscaler/SYSTEM_DOMAIN)) shield-url: https://shield.((/app-autoscaler/SYSTEM_DOMAIN)) release: shield name: postgres_autoscaler networks:
  • name: dynamic persistent_disk: 10000 stemcell: default update: serial: true vm_type: small
• azs:
  • z1 instances: 1 jobs:
  • name: scalingengine properties: autoscaler: cf: api: https://api.((system_domain)) client_id: ((cf_client_id)) grant_type: client_credentials secret: ((cf_client_secret)) skip_ssl_validation: false policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 scalingengine: ca_cert: ((scalingengine_ca.ca)) defaultCoolDownSecs: 300 health: password: ((autoscaler_scalingengine_health_password)) port: 6204 username: scalingengine http_client_timeout: 60s lockSize: 32 logging: level: info server: port: 6104 server_cert: ((scalingengine_server.certificate)) server_key: ((scalingengine_server.private_key)) scalingengine_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) scalingengine_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 scheduler_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) scheduler_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • name: scheduler properties: autoscaler: policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) scheduler: ca_cert: ((scheduler_ca.ca)) health: basicAuthEnabled: true password: ((autoscaler_scheduler_health_password)) port: 6202 username: scheduler http_client_timeout: 60 job_reschedule_interval_millisecond: 10000 job_reschedule_maxcount: 6 notification_reschedule_maxcount: 3 port: 6102 scaling_engine: ca_cert: ((scalingengine_ca.ca)) client_cert: ((scalingengine_client.certificate)) client_key: ((scalingengine_client.private_key)) server_cert: ((scheduler_server.certificate)) server_key: ((scheduler_server.private_key)) scheduler_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) release: app-autoscaler
  • name: operator properties: autoscaler: appmetrics_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) appmetrics_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 cf: api: https://api.((system_domain)) client_id: ((cf_client_id)) grant_type: client_credentials secret: ((cf_client_secret)) skip_ssl_validation: false instancemetrics_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) instancemetrics_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 lock_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) lock_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 operator: app_sync_interval: 24h db_lock: retry_interval: 5s ttl: 15s health: password: ((autoscaler_operator_health_password)) port: 6208 username: operator http_client_timeout: 60s logging: level: info scaling_engine: ca_cert: ((scalingengine_ca.ca)) client_cert: ((scalingengine_client.certificate)) client_key: ((scalingengine_client.private_key)) port: 6104 scheduler: ca_cert: ((scheduler_ca.ca)) client_cert: ((scheduler_client.certificate)) client_key: ((scheduler_client.private_key)) port: 6102 policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 scalingengine_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) scalingengine_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • consumes: nats: deployment: cf from: nats name: route_registrar properties: route_registrar: routes:
    • name: autoscaler_scalingengine_health port: 6204 registration_interval: 20s tags: component: autoscaler_scalingengine_health uris:
      • autoscaler-scalingengine.((system_domain))
    • name: autoscaler_operator_health port: 6208 registration_interval: 20s tags: component: autoscaler_operator_health uris:
      • autoscaler-operator.((system_domain))
    • name: autoscaler_scheduler_health port: 6202 registration_interval: 20s tags: component: autoscaler_scheduler_health uris:
      • autoscaler-scheduler.((system_domain)) release: routing name: asactors networks:
  • name: dynamic stemcell: default update: max_in_flight: 1 serial: true vm_type: xsmall
• azs:
  • z1 instances: 1 jobs:
  • name: metricsserver properties: autoscaler: instancemetrics_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) instancemetrics_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 metricsserver: ca_cert: ((metricsserver_server.ca)) collector: collect_interval: 60s envelope_channel_size: 1000 envelope_processor_count: 5 keep_alive_time: 60s metric_cache_size_per_app: 1000 metric_channel_size: 1000 persist_metrics: true port: 7103 refresh_interval: 60s save_interval: 5s health: password: ((autoscaler_metricsserver_health_password)) port: 6303 username: metricsserver http_client_timeout: 60s logging: level: info server: port: 6103 server_cert: ((metricsserver_server.certificate)) server_key: ((metricsserver_server.private_key)) policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • name: eventgenerator properties: autoscaler: appmetrics_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) appmetrics_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 eventgenerator: aggregator: aggregator_execute_interval: 40s app_metric_channel_size: 1000 app_monitor_channel_size: 200 metric_poller_count: 20 policy_poller_interval: 60s save_interval: 5s ca_cert: ((eventgenerator_ca.ca)) circuitBreaker: back_off_initial_interval: 5m back_off_max_interval: 120m consecutive_failure_count: 5 defaultBreachDurationSecs: 120 defaultStatWindowSecs: 120 evaluator: evaluation_manager_execute_interval: 60s evaluator_count: 20 trigger_array_channel_size: 200 health: password: ((autoscaler_eventgenerator_health_password)) port: 6205 username: eventgenerator http_client_timeout: 60s logging: level: info metricscollector: ca_cert: ((metricsserver_ca.ca)) client_cert: ((metricsserver_client.certificate)) client_key: ((metricsserver_client.private_key)) host: metricsserver.service.cf.internal port: 6103 scaling_engine: ca_cert: ((scalingengine_ca.ca)) client_cert: ((scalingengine_client.certificate)) client_key: ((scalingengine_client.private_key)) host: scalingengine.service.cf.internal port: 6104 server: port: 6105 server_cert: ((eventgenerator_server.certificate)) server_key: ((eventgenerator_server.private_key)) policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • consumes: nats: deployment: cf from: nats name: route_registrar properties: route_registrar: routes:
    • name: autoscaler_eventgenerator_health port: 6205 registration_interval: 20s tags: component: autoscaler_eventgenerator_health uris:
      • autoscaler-eventgenerator.((system_domain))
    • name: autoscaler_metricsserver_health port: 6303 registration_interval: 20s tags: component: autoscaler_metricsserver_health uris:
      • autoscaler-metricsserver.((system_domain)) release: routing name: asmetrics networks:
  • name: dynamic stemcell: default update: max_in_flight: 1 serial: true vm_type: small
• azs:
  • z1 instances: 1 jobs:
  • name: metricsgateway properties: autoscaler: metricsgateway: app_manager: app_refresh_interval: 5s emitter: buffer_size: 500 handshake_timeout: 1s keep_alive_interval: 5s max_close_retry_count: 3 max_setup_retry_count: 3 metricsserver_client: ca_cert: ((metricsserver_client.ca)) cert: ((metricsserver_client.certificate)) key: ((metricsserver_client.private_key)) retry_delay: 1s envelop_chan_size: 1000 health: password: ((autoscaler_metricsgateway_health_password)) port: 6503 username: metricsgateway logging: level: info nozzle: loggregator_rlp_tls: ca_cert: ((/cf/loggregator_ca.certificate)) cert: ((/cf/loggregator_tls_rlp.certificate)) key: ((/cf/loggregator_tls_rlp.private_key)) rlp_addr: reverse-log-proxy.service.cf.internal:8082 shard_id: CF_AUTOSCALER nozzle_count: 3 policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • consumes: nats: deployment: cf from: nats name: route_registrar properties: route_registrar: routes:
    • name: autoscaler_metricsgateway_health port: 6503 registration_interval: 20s tags: component: autoscaler_metricsgateway_health uris:
      • autoscaler-metricsgateway.((system_domain)) release: routing name: asnozzle networks:
  • name: dynamic stemcell: default update: max_in_flight: 1 serial: true vm_type: small
• azs:
  • z1 instances: 1 jobs:
  • name: golangapiserver properties: autoscaler: apiserver: broker: password: ((autoscaler_service_broker_password)) server: catalog: services:
    • bindable: true description: Automatically increase or decrease the number of application instances based on a policy you define. id: autoscaler-guid name: autoscaler plans:
      • description: This is the free service plan for the Auto-Scaling service. id: 4223a80e-24f5-4374-a101-99fb45675acc name: autoscaler-free-plan dashboard_redirect_uri: "" port: 6102 username: autoscaler_service_broker_user event_generator: ca_cert: ((eventgenerator_ca.ca)) client_cert: ((eventgenerator_client.certificate)) client_key: ((eventgenerator_client.private_key)) logging: level: info metrics_collector: ca_cert: ((metricsserver_ca.ca)) client_cert: ((metricsserver_client.certificate)) client_key: ((metricsserver_client.private_key)) host: metricsserver.service.cf.internal port: 6103 metrics_forwarder: host: autoscalermetrics.((system_domain)) public_api: server: port: 6101 scaling_engine: ca_cert: ((scalingengine_ca.ca)) client_cert: ((scalingengine_client.certificate)) client_key: ((scalingengine_client.private_key)) scheduler: ca_cert: ((scheduler_ca.ca)) client_cert: ((scheduler_client.certificate)) client_key: ((scheduler_client.private_key)) use_buildin_mode: false binding_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
        • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
        • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) binding_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 cf: api: https://api.((system_domain)) client_id: ((cf_client_id)) grant_type: client_credentials secret: ((cf_client_secret)) skip_ssl_validation: false policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
        • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
        • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • name: metricsforwarder properties: autoscaler: metricsforwarder: cache_cleanup_interval: 6h cache_ttl: 900s health: password: ((autoscaler_metricsforwarder_health_password)) port: 6403 username: metricsforwarder logging: level: info loggregator: metron_address: 127.0.0.1:3458 tls: ca_cert: ((/cf/loggregator_tls_agent.ca)) cert: ((/cf/loggregator_tls_agent.certificate)) key: ((/cf/loggregator_tls_agent.private_key)) policy_poller_interval: 60s require_consule: false server: port: 6201 policy_db: connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 databases:
    • name: autoscaler tag: default db_scheme: postgres port: 5432 roles:
    • name: postgres password: ((database_password)) tag: default sslmode: verify-full tls: ca: ((postgres_ca.ca)) certificate: ((postgres_server.certificate)) private_key: ((postgres_server.private_key)) policy_db_connection_config: connection_max_lifetime: 60s max_idle_connections: 10 max_open_connections: 100 release: app-autoscaler
  • consumes: nats: deployment: cf from: nats name: route_registrar properties: route_registrar: routes:
    • name: api_server port: 6101 registration_interval: 20s tags: component: api_server uris:
      • autoscaler.((system_domain))
    • name: autoscaler_service_broker port: 6102 registration_interval: 20s tags: component: autoscaler_service_broker uris:
      • autoscalerservicebroker.((system_domain))
    • name: autoscaler_metrics_forwarder port: 6201 registration_interval: 20s tags: component: autoscaler_metrics_forwarder uris:
      • autoscalermetrics.((system_domain))
    • name: autoscaler_metricsforwarder_health port: 6403 registration_interval: 20s tags: component: autoscaler_metricsforwarder_health uris:
      • autoscaler-metricsforwarder.((system_domain)) release: routing
  • consumes: doppler: instances:
    • address: doppler.service.cf.internal name: loggregator_agent properties: loggregator: tls: agent: cert: ((/cf/loggregator_tls_agent.certificate)) key: ((/cf/loggregator_tls_agent.private_key)) ca_cert: ((/cf/loggregator_ca.certificate)) release: loggregator-agent name: asapi networks:
  • name: dynamic stemcell: default update: max_in_flight: 1 serial: true vm_type: xsmall name: app-autoscaler releases:
• name: app-autoscaler version: latest
• name: postgres sha1: 24d2e2887a45258b71bc40577c0f406180e47701 url: https://bosh.io/d/github.com/cloudfoundry/postgres-release?v=29 version: "29"
• name: bosh-dns-aliases sha1: b0d0a0350ed87f1ded58b2ebb469acea0e026ccc url: https://bosh.io/d/github.com/cloudfoundry/bosh-dns-aliases-release?v=0.0.3 version: 0.0.3
• name: routing sha1: edb2f822ca5bbe579a179abfa4b053dc9c99224f url: https://bosh.io/d/github.com/cloudfoundry/routing-release?v=0.223.0 version: 0.223.0
• name: loggregator-agent sha1: bf723af956a61c7b51db0ba535c871bad24dd289 url: https://bosh.io/d/github.com/cloudfoundry/loggregator-agent-release?v=3.9 version: "3.9"
• name: bpm sha1: sha256:031cf5600158f44762c747a1f8648b7cebca876d4297dd9e2ce3b51b36ccd476 url: https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.1.9 version: 1.1.9
• name: shield sha1: fe7eb0f995fddb171a3766ce5ee05d58f42c8512 url: https://github.com/shieldproject/shield-boshrelease/releases/download/v8.7.4/shield-8.7.4.tgz version: 8.7.4 stemcells:
• alias: default os: ubuntu-bionic version: latest update: canaries: 1 canary_watch_time: 1000-300000 max_in_flight: 1 update_watch_time: 1000-300000 variables:
• name: database_password type: password
• name: autoscaler_service_broker_password type: password
• name: autoscaler_scheduler_health_password type: password
• name: autoscaler_eventgenerator_health_password type: password
• name: autoscaler_metricsforwarder_health_password type: password
• name: autoscaler_metricsgateway_health_password type: password
• name: autoscaler_metricsserver_health_password type: password
• name: autoscaler_operator_health_password type: password
• name: autoscaler_scalingengine_health_password type: password
• name: scalingengine_ca options: common_name: scalingengineCA is_ca: true type: certificate
• name: scalingengine_server options: alternative_names:
  • scalingengine.service.cf.internal ca: scalingengine_ca common_name: scalingengine.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: scalingengine_client options: ca: scalingengine_ca common_name: scalingengine client extended_key_usage:
  • client_auth type: certificate
• name: eventgenerator_ca options: common_name: eventgeneratorCA is_ca: true type: certificate
• name: eventgenerator_server options: alternative_names:
  • eventgenerator.service.cf.internal ca: eventgenerator_ca common_name: eventgenerator.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: eventgenerator_client options: ca: eventgenerator_ca common_name: eventgenerator client extended_key_usage:
  • client_auth type: certificate
• name: apiserver_ca options: common_name: apiserverCA is_ca: true type: certificate
• name: apiserver_server options: alternative_names:
  • apiserver.service.cf.internal ca: apiserver_ca common_name: apiserver.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: apiserver_public_ca options: common_name: apiserverpublicCA is_ca: true type: certificate
• name: apiserver_public_server options: alternative_names:
  • autoscaler.((system_domain)) ca: apiserver_public_ca common_name: autoscaler.((system_domain)) extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: apiserver_client options: ca: apiserver_ca common_name: apiserver client extended_key_usage:
  • client_auth type: certificate
• name: servicebroker_ca options: common_name: servicebrokerCA is_ca: true type: certificate
• name: servicebroker_server options: alternative_names:
  • servicebroker.service.cf.internal ca: servicebroker_ca common_name: servicebroker.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: servicebroker_client options: ca: servicebroker_ca common_name: servicebroker client extended_key_usage:
  • client_auth type: certificate
• name: servicebroker_public_ca options: common_name: servicebrokerCA is_ca: true type: certificate
• name: servicebroker_public_server options: alternative_names:
  • autoscalerservicebroker.((system_domain)) ca: servicebroker_public_ca common_name: autoscalerservicebroker.((system_domain)) extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: scheduler_ca options: common_name: schedulerCA is_ca: true type: certificate
• name: scheduler_server options: alternative_names:
  • autoscalerscheduler.service.cf.internal ca: scheduler_ca common_name: autoscalerscheduler.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: scheduler_client options: ca: scheduler_ca common_name: scheduler client extended_key_usage:
  • client_auth type: certificate
• name: metricsserver_ca options: common_name: metricsserverCA is_ca: true type: certificate
• name: metricsserver_server options: alternative_names:
  • metricsserver.service.cf.internal
  • '*.asmetrics.default.app-autoscaler.bosh'
  • '*.asmetrics.dynamic.app-autoscaler.bosh' ca: metricsserver_ca common_name: metricsserver.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate
• name: metricsserver_client options: ca: metricsserver_ca common_name: metricsserver client extended_key_usage:
  • client_auth type: certificate
• name: postgres_ca options: common_name: postgresCA is_ca: true type: certificate
• name: postgres_server options: alternative_names:
  • autoscalerpostgres.service.cf.internal ca: postgres_ca common_name: autoscalerpostgres.service.cf.internal extended_key_usage:
  • client_auth
  • server_auth type: certificate `

Any clue will be much appreciated.

Thanks a lot,

[silvestre]

Hi @mmavrudiev ,

as discussed on slack, you should investigate

While debugging I was the following errors in asapi node - loggregator_agent logs: 2022/03/11 13:17:44.042103 failed to connect: unable to lookup a log consumer 2022/03/11 13:17:44.042158 failed to connect: unable to lookup a log consumer

The loggregator_agent is used to inject the custom metrics into the loggerator, and is a reused component documented at https://github.com/cloudfoundry/loggregator-agent-release/blob/main/docs/loggregator-agent.md.

[silvestre]

This issue seems to have been resolved - please reopen if there's still something to do.