VM Agent: slow proceccing and insufficient resource utilization

Can't figure out what I'm doing wrong.

What I have:

One single instance of vmagent with CPU: 14 and Memory: 16Gi
~1400 targets from prometheus SD.
Very slow scraping: after 10 min only ~450 targets were scrapped once
Recource utilization: ~100m CPU, <1Gi memory

What I expected Resources would be utilized with proportionally high performance.

Metrics

Expand

``` process_cpu_cores_available 14 process_memory_limit_bytes 17179869184 vm_cache_chars_current{type="promql/regexp"} 0 vm_cache_chars_max{type="promql/regexp"} 1000000 vm_cache_entries{type="promql/regexp"} 0 vm_cache_misses_total{type="promql/regexp"} 0 vm_cache_requests_total{type="promql/regexp"} 0 vm_concurrent_insert_capacity 128 vm_concurrent_insert_current 0 vm_concurrent_insert_limit_reached_total 0 vm_concurrent_insert_limit_timeout_total 0 vm_filestream_buffered_read_bytes_total 0 vm_filestream_buffered_read_calls_total 0 vm_filestream_buffered_write_calls_total 1 vm_filestream_buffered_written_bytes_total 0 vm_filestream_read_duration_seconds_total 0 vm_filestream_readers 1 vm_filestream_real_read_bytes_total 0 vm_filestream_real_read_calls_total 0 vm_filestream_real_write_calls_total 0 vm_filestream_real_written_bytes_total 0 vm_filestream_write_duration_seconds_total 4.4e-07 vm_filestream_writers 1 vm_fs_read_bytes_total 0 vm_fs_read_calls_total 0 vm_fs_readers 0 vm_http_conn_timeout_closed_conns_total 0 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="1.668e-05...1.896e-05"} 4 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="1.896e-05...2.154e-05"} 1 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="1.468e-04...1.668e-04"} 1 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="2.783e-04...3.162e-04"} 1 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="1.668e-03...1.896e-03"} 7 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="2.154e-03...2.448e-03"} 1 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="2.448e-03...2.783e-03"} 1 vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="6.813e-03...7.743e-03"} 1 vm_http_request_duration_seconds_sum{path="/metrics"} 0.024263377000000003 vm_http_request_duration_seconds_count{path="/metrics"} 17 vm_http_request_errors_total{path="*", reason="unsupported"} 0 vm_http_requests_all_total 53 vm_http_requests_total{path="/debug/pprof/"} 0 vm_http_requests_total{path="/debug/pprof/cmdline"} 0 vm_http_requests_total{path="/debug/pprof/default"} 0 vm_http_requests_total{path="/debug/pprof/mutex"} 0 vm_http_requests_total{path="/debug/pprof/profile"} 0 vm_http_requests_total{path="/debug/pprof/symbol"} 0 vm_http_requests_total{path="/debug/pprof/trace"} 0 vm_http_requests_total{path="/favicon.ico"} 21 vm_http_requests_total{path="/metrics"} 18 vm_ingestserver_request_errors_total{type="graphite", name="write", net="tcp"} 0 vm_ingestserver_request_errors_total{type="graphite", name="write", net="udp"} 0 vm_ingestserver_request_errors_total{type="influx", name="write", net="tcp"} 0 vm_ingestserver_request_errors_total{type="influx", name="write", net="udp"} 0 vm_ingestserver_request_errors_total{type="opentsdb", name="write", net="tcp"} 0 vm_ingestserver_request_errors_total{type="opentsdb", name="write", net="udp"} 0 vm_ingestserver_request_errors_total{type="opentsdbhttp", name="write", net="tcp"} 0 vm_ingestserver_requests_total{type="graphite", name="write", net="tcp"} 0 vm_ingestserver_requests_total{type="graphite", name="write", net="udp"} 0 vm_ingestserver_requests_total{type="influx", name="write", net="tcp"} 0 vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 0 vm_ingestserver_requests_total{type="opentsdb", name="write", net="tcp"} 0 vm_ingestserver_requests_total{type="opentsdb", name="write", net="udp"} 0 vm_ingestserver_requests_total{type="opentsdbhttp", name="write", net="tcp"} 0 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/app/vmagent/main.go:103"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/app/vmagent/main.go:130"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/app/vmagent/remotewrite/client.go:169"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/httpserver/httpserver.go:96"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/httpserver/httpserver.go:97"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/logger/flag.go:12"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/logger/flag.go:13"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/logger/flag.go:20"} 6 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/memory/memory.go:42"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/persistentqueue/fastqueue.go:59"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/promscrape/config.go:120"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/promscrape/config.go:126"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:501"} 3 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/promscrape/discovery/kubernetes/api_watcher.go:633"} 3 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/promscrape/scraper.go:106"} 1 vm_log_messages_total{app_version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", level="info", location="VictoriaMetrics/lib/promscrape/scraper.go:403"} 2 vm_nfs_dir_remove_failed_attempts_total 0 vm_nfs_pending_dirs_to_remove 0 vm_persistentqueue_blocks_dropped_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_blocks_read_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_blocks_written_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_bytes_dropped_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_bytes_pending{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_bytes_read_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_bytes_written_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE"} 0 vm_persistentqueue_read_duration_seconds_total 0 vm_persistentqueue_write_duration_seconds_total 0 vm_promscrape_active_scrapers{type="azure_sd_configs"} 0 vm_promscrape_active_scrapers{type="consul_sd_configs"} 0 vm_promscrape_active_scrapers{type="digitalocean_sd_configs"} 0 vm_promscrape_active_scrapers{type="dns_sd_configs"} 0 vm_promscrape_active_scrapers{type="docker_sd_configs"} 0 vm_promscrape_active_scrapers{type="dockerswarm_sd_configs"} 0 vm_promscrape_active_scrapers{type="ec2_sd_configs"} 0 vm_promscrape_active_scrapers{type="eureka_sd_configs"} 0 vm_promscrape_active_scrapers{type="file_sd_configs"} 0 vm_promscrape_active_scrapers{type="gce_sd_configs"} 0 vm_promscrape_active_scrapers{type="http_sd_configs"} 0 vm_promscrape_active_scrapers{type="kubernetes_sd_configs"} 1403 vm_promscrape_active_scrapers{type="openstack_sd_configs"} 0 vm_promscrape_active_scrapers{type="static_configs"} 1 vm_promscrape_active_scrapers{type="yandexcloud_sd_configs"} 0 vm_promscrape_config_changes_total{type="azure_sd_configs"} 0 vm_promscrape_config_changes_total{type="consul_sd_configs"} 0 vm_promscrape_config_changes_total{type="digitalocean_sd_configs"} 0 vm_promscrape_config_changes_total{type="dns_sd_configs"} 0 vm_promscrape_config_changes_total{type="docker_sd_configs"} 0 vm_promscrape_config_changes_total{type="dockerswarm_sd_configs"} 0 vm_promscrape_config_changes_total{type="ec2_sd_configs"} 0 vm_promscrape_config_changes_total{type="eureka_sd_configs"} 0 vm_promscrape_config_changes_total{type="file_sd_configs"} 0 vm_promscrape_config_changes_total{type="gce_sd_configs"} 0 vm_promscrape_config_changes_total{type="http_sd_configs"} 0 vm_promscrape_config_changes_total{type="kubernetes_sd_configs"} 1403 vm_promscrape_config_changes_total{type="openstack_sd_configs"} 0 vm_promscrape_config_changes_total{type="static_configs"} 1 vm_promscrape_config_changes_total{type="yandexcloud_sd_configs"} 0 vm_promscrape_config_reloads_total 0 vm_promscrape_conn_bytes_read_total 596611293 vm_promscrape_conn_bytes_written_total 111278 vm_promscrape_conn_read_errors_total 0 vm_promscrape_conn_reads_total 15936 vm_promscrape_conn_write_errors_total 0 vm_promscrape_conn_writes_total 551 vm_promscrape_conns 551 vm_promscrape_dial_errors_total 0 vm_promscrape_dials_total 551 vm_promscrape_discovery_consul_service_watchers 0 vm_promscrape_discovery_consul_service_watchers_created_total 0 vm_promscrape_discovery_consul_service_watchers_stopped_total 0 vm_promscrape_discovery_kubernetes_group_watchers 1 vm_promscrape_discovery_kubernetes_objects_added_total{role="endpoints"} 39 vm_promscrape_discovery_kubernetes_objects_added_total{role="pod"} 812 vm_promscrape_discovery_kubernetes_objects_added_total{role="service"} 39 vm_promscrape_discovery_kubernetes_objects_removed_total{role="endpoints"} 0 vm_promscrape_discovery_kubernetes_objects_removed_total{role="pod"} 1 vm_promscrape_discovery_kubernetes_objects_removed_total{role="service"} 0 vm_promscrape_discovery_kubernetes_objects_updated_total{role="endpoints"} 3 vm_promscrape_discovery_kubernetes_objects_updated_total{role="pod"} 4 vm_promscrape_discovery_kubernetes_objects_updated_total{role="service"} 0 vm_promscrape_discovery_kubernetes_objects{role="endpoints"} 39 vm_promscrape_discovery_kubernetes_objects{role="pod"} 811 vm_promscrape_discovery_kubernetes_objects{role="service"} 39 vm_promscrape_discovery_kubernetes_scrape_works{role="endpoints"} 1403 vm_promscrape_discovery_kubernetes_stale_resource_versions_total{role="endpoints"} 0 vm_promscrape_discovery_kubernetes_stale_resource_versions_total{role="pod"} 0 vm_promscrape_discovery_kubernetes_stale_resource_versions_total{role="service"} 0 vm_promscrape_discovery_kubernetes_subscribers{role="endpoints",status="pending"} 0 vm_promscrape_discovery_kubernetes_subscribers{role="endpoints",status="working"} 2 vm_promscrape_discovery_kubernetes_url_watchers{role="endpoints"} 1 vm_promscrape_discovery_kubernetes_url_watchers{role="pod"} 1 vm_promscrape_discovery_kubernetes_url_watchers{role="service"} 1 vm_promscrape_discovery_requests_total 0 vm_promscrape_discovery_retries_total 0 vm_promscrape_discoveryutils_configmap_entries_count 0 vm_promscrape_max_scrape_size_exceeded_errors_total 0 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.468e-05...1.668e-05"} 23 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.668e-05...1.896e-05"} 10 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.896e-05...2.154e-05"} 3 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.154e-05...2.448e-05"} 9 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.448e-05...2.783e-05"} 2 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.783e-05...3.162e-05"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.594e-05...4.084e-05"} 2 vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.084e-05...4.642e-05"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.743e-05...8.799e-05"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.154e-04...2.448e-04"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.162e-04...3.594e-04"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.594e-04...4.084e-04"} 66 vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.084e-04...4.642e-04"} 48 vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.642e-04...5.275e-04"} 2 vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.995e-04...6.813e-04"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.468e-03...1.668e-03"} 4 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.668e-03...1.896e-03"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.995e-03...6.813e-03"} 5 vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.813e-03...7.743e-03"} 50 vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.743e-03...8.799e-03"} 107 vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.799e-03...1.000e-02"} 39 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.000e-02...1.136e-02"} 7 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.136e-02...1.292e-02"} 28 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.292e-02...1.468e-02"} 51 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.468e-02...1.668e-02"} 16 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.668e-02...1.896e-02"} 24 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.896e-02...2.154e-02"} 13 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.154e-02...2.448e-02"} 7 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.448e-02...2.783e-02"} 9 vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.783e-02...3.162e-02"} 4 vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.162e-02...3.594e-02"} 3 vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.594e-02...4.084e-02"} 2 vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.642e-02...5.275e-02"} 4 vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.275e-02...5.995e-02"} 2 vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.995e-02...6.813e-02"} 1 vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.813e-02...7.743e-02"} 2 vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.000e-01...1.136e-01"} 1 vm_promscrape_push_data_duration_seconds_sum 5.108307740000003 vm_promscrape_push_data_duration_seconds_count 551 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.896e-03...2.154e-03"} 21 vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.783e-03...3.162e-03"} 20 vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.594e-03...4.084e-03"} 8 vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.642e-03...5.275e-03"} 3 vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.813e-03...7.743e-03"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.743e-03...8.799e-03"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.799e-03...1.000e-02"} 12 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.000e-02...1.136e-02"} 19 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.136e-02...1.292e-02"} 13 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.292e-02...1.468e-02"} 25 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.468e-02...1.668e-02"} 24 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.668e-02...1.896e-02"} 10 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.896e-02...2.154e-02"} 9 vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.154e-02...2.448e-02"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.783e-02...3.162e-02"} 2 vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.594e-02...4.084e-02"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.084e-02...4.642e-02"} 3 vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.642e-02...5.275e-02"} 5 vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.275e-02...5.995e-02"} 6 vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.995e-02...6.813e-02"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.813e-02...7.743e-02"} 12 vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.743e-02...8.799e-02"} 83 vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.799e-02...1.000e-01"} 171 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.000e-01...1.136e-01"} 54 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.136e-01...1.292e-01"} 19 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.292e-01...1.468e-01"} 9 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.468e-01...1.668e-01"} 2 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.668e-01...1.896e-01"} 4 vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.896e-01...2.154e-01"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.154e-01...2.448e-01"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.448e-01...2.783e-01"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.594e-01...4.084e-01"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.084e-01...4.642e-01"} 2 vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.642e-01...5.275e-01"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.995e-01...6.813e-01"} 2 vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.743e-01...8.799e-01"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.799e-01...1.000e+00"} 1 vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.162e+00...3.594e+00"} 1 vm_promscrape_scrape_duration_seconds_sum 45.67800000000012 vm_promscrape_scrape_duration_seconds_count 551 vm_promscrape_scrape_requests_total 551 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="5.275e+03...5.995e+03"} 51 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.154e+04...2.448e+04"} 1 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="7.743e+04...8.799e+04"} 1 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.136e+05...1.292e+05"} 1 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.292e+05...1.468e+05"} 123 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="7.743e+05...8.799e+05"} 3 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="8.799e+05...1.000e+06"} 4 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.000e+06...1.136e+06"} 11 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.136e+06...1.292e+06"} 1 vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.468e+06...1.668e+06"} 355 vm_promscrape_scrape_response_size_bytes_sum 596364182 vm_promscrape_scrape_response_size_bytes_count 551 vm_promscrape_scrape_retries_total 0 vm_promscrape_scraped_samples_bucket{vmrange="4.084e+01...4.642e+01"} 51 vm_promscrape_scraped_samples_bucket{vmrange="1.896e+02...2.154e+02"} 1 vm_promscrape_scraped_samples_bucket{vmrange="4.642e+02...5.275e+02"} 1 vm_promscrape_scraped_samples_bucket{vmrange="1.000e+03...1.136e+03"} 6 vm_promscrape_scraped_samples_bucket{vmrange="1.136e+03...1.292e+03"} 118 vm_promscrape_scraped_samples_bucket{vmrange="4.084e+03...4.642e+03"} 1 vm_promscrape_scraped_samples_bucket{vmrange="4.642e+03...5.275e+03"} 2 vm_promscrape_scraped_samples_bucket{vmrange="5.275e+03...5.995e+03"} 10 vm_promscrape_scraped_samples_bucket{vmrange="5.995e+03...6.813e+03"} 6 vm_promscrape_scraped_samples_bucket{vmrange="7.743e+03...8.799e+03"} 189 vm_promscrape_scraped_samples_bucket{vmrange="8.799e+03...1.000e+04"} 166 vm_promscrape_scraped_samples_sum 3372170 vm_promscrape_scraped_samples_count 551 vm_promscrape_scrapers_started_total{type="azure_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="consul_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="digitalocean_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="dns_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="docker_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="dockerswarm_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="ec2_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="eureka_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="file_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="gce_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="http_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="kubernetes_sd_configs"} 1403 vm_promscrape_scrapers_started_total{type="openstack_sd_configs"} 0 vm_promscrape_scrapers_started_total{type="static_configs"} 1 vm_promscrape_scrapers_started_total{type="yandexcloud_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="azure_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="consul_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="digitalocean_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="dns_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="docker_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="dockerswarm_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="ec2_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="eureka_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="file_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="gce_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="http_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="kubernetes_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="openstack_sd_configs"} 0 vm_promscrape_scrapers_stopped_total{type="static_configs"} 0 vm_promscrape_scrapers_stopped_total{type="yandexcloud_sd_configs"} 0 vm_promscrape_scrapes_failed_total 0 vm_promscrape_scrapes_gunzip_failed_total 0 vm_promscrape_scrapes_gunziped_total 0 vm_promscrape_scrapes_skipped_by_sample_limit_total 0 vm_promscrape_scrapes_timed_out_total 0 vm_promscrape_scrapes_total{status_code="200"} 551 vm_promscrape_service_discovery_duration_seconds_bucket{type="kubernetes_sd_configs",vmrange="2.448e-02...2.783e-02"} 20 vm_promscrape_service_discovery_duration_seconds_bucket{type="kubernetes_sd_configs",vmrange="3.162e-02...3.594e-02"} 1 vm_promscrape_service_discovery_duration_seconds_bucket{type="kubernetes_sd_configs",vmrange="3.594e-02...4.084e-02"} 1 vm_promscrape_service_discovery_duration_seconds_bucket{type="kubernetes_sd_configs",vmrange="4.084e-02...4.642e-02"} 1 vm_promscrape_service_discovery_duration_seconds_bucket{type="kubernetes_sd_configs",vmrange="1.668e-01...1.896e-01"} 1 vm_promscrape_service_discovery_duration_seconds_sum{type="kubernetes_sd_configs"} 0.7991676989999998 vm_promscrape_service_discovery_duration_seconds_count{type="kubernetes_sd_configs"} 24 vm_promscrape_service_discovery_duration_seconds_bucket{type="static_configs",vmrange="1.136e-03...1.292e-03"} 1 vm_promscrape_service_discovery_duration_seconds_sum{type="static_configs"} 0.001268737 vm_promscrape_service_discovery_duration_seconds_count{type="static_configs"} 1 vm_promscrape_stale_samples_created_total 0 vm_promscrape_targets{type="azure_sd_configs", status="down"} 0 vm_promscrape_targets{type="azure_sd_configs", status="up"} 0 vm_promscrape_targets{type="consul_sd_configs", status="down"} 0 vm_promscrape_targets{type="consul_sd_configs", status="up"} 0 vm_promscrape_targets{type="digitalocean_sd_configs", status="down"} 0 vm_promscrape_targets{type="digitalocean_sd_configs", status="up"} 0 vm_promscrape_targets{type="dns_sd_configs", status="down"} 0 vm_promscrape_targets{type="dns_sd_configs", status="up"} 0 vm_promscrape_targets{type="docker_sd_configs", status="down"} 0 vm_promscrape_targets{type="docker_sd_configs", status="up"} 0 vm_promscrape_targets{type="dockerswarm_sd_configs", status="down"} 0 vm_promscrape_targets{type="dockerswarm_sd_configs", status="up"} 0 vm_promscrape_targets{type="ec2_sd_configs", status="down"} 0 vm_promscrape_targets{type="ec2_sd_configs", status="up"} 0 vm_promscrape_targets{type="eureka_sd_configs", status="down"} 0 vm_promscrape_targets{type="eureka_sd_configs", status="up"} 0 vm_promscrape_targets{type="file_sd_configs", status="down"} 0 vm_promscrape_targets{type="file_sd_configs", status="up"} 0 vm_promscrape_targets{type="gce_sd_configs", status="down"} 0 vm_promscrape_targets{type="gce_sd_configs", status="up"} 0 vm_promscrape_targets{type="http_sd_configs", status="down"} 0 vm_promscrape_targets{type="http_sd_configs", status="up"} 0 vm_promscrape_targets{type="kubernetes_sd_configs", status="down"} 852 vm_promscrape_targets{type="kubernetes_sd_configs", status="up"} 551 vm_promscrape_targets{type="openstack_sd_configs", status="down"} 0 vm_promscrape_targets{type="openstack_sd_configs", status="up"} 0 vm_promscrape_targets{type="static_configs", status="down"} 1 vm_promscrape_targets{type="static_configs", status="up"} 0 vm_promscrape_targets{type="yandexcloud_sd_configs", status="down"} 0 vm_promscrape_targets{type="yandexcloud_sd_configs", status="up"} 0 vm_protoparser_blocks_read_total{type="native"} 0 vm_protoparser_parse_errors_total{type="native"} 0 vm_protoparser_process_errors_total{type="native"} 0 vm_protoparser_read_calls_total{type="csvimport"} 0 vm_protoparser_read_calls_total{type="datadog"} 0 vm_protoparser_read_calls_total{type="graphite"} 0 vm_protoparser_read_calls_total{type="influx"} 0 vm_protoparser_read_calls_total{type="native"} 0 vm_protoparser_read_calls_total{type="opentsdb"} 0 vm_protoparser_read_calls_total{type="opentsdbhttp"} 0 vm_protoparser_read_calls_total{type="prometheus"} 0 vm_protoparser_read_calls_total{type="promremotewrite"} 0 vm_protoparser_read_calls_total{type="vmimport"} 0 vm_protoparser_read_errors_total{type="csvimport"} 0 vm_protoparser_read_errors_total{type="datadog"} 0 vm_protoparser_read_errors_total{type="graphite"} 0 vm_protoparser_read_errors_total{type="influx"} 0 vm_protoparser_read_errors_total{type="native"} 0 vm_protoparser_read_errors_total{type="opentsdb"} 0 vm_protoparser_read_errors_total{type="opentsdbhttp"} 0 vm_protoparser_read_errors_total{type="prometheus"} 0 vm_protoparser_read_errors_total{type="promremotewrite"} 0 vm_protoparser_read_errors_total{type="vmimport"} 0 vm_protoparser_rows_read_total{type="csvimport"} 0 vm_protoparser_rows_read_total{type="datadog"} 0 vm_protoparser_rows_read_total{type="graphite"} 0 vm_protoparser_rows_read_total{type="influx"} 0 vm_protoparser_rows_read_total{type="native"} 0 vm_protoparser_rows_read_total{type="opentsdb"} 0 vm_protoparser_rows_read_total{type="opentsdbhttp"} 0 vm_protoparser_rows_read_total{type="prometheus"} 0 vm_protoparser_rows_read_total{type="promremotewrite"} 0 vm_protoparser_rows_read_total{type="promscrape"} 3372170 vm_protoparser_rows_read_total{type="vmimport"} 0 vm_protoparser_unmarshal_errors_total{type="datadog"} 0 vm_protoparser_unmarshal_errors_total{type="opentsdbhttp"} 0 vm_protoparser_unmarshal_errors_total{type="promremotewrite"} 0 vm_rows_invalid_total{type="csvimport"} 0 vm_rows_invalid_total{type="graphite"} 0 vm_rows_invalid_total{type="influx"} 0 vm_rows_invalid_total{type="opentsdb"} 0 vm_rows_invalid_total{type="opentsdbhttp"} 0 vm_rows_invalid_total{type="prometheus"} 0 vm_rows_invalid_total{type="vmimport"} 0 vm_tcplistener_accepts_total{name="http", addr=":8429"} 17 vm_tcplistener_conns{name="http", addr=":8429"} 2 vm_tcplistener_errors_total{name="http", addr=":8429", type="accept"} 0 vm_tcplistener_errors_total{name="http", addr=":8429", type="close"} 0 vm_tcplistener_errors_total{name="http", addr=":8429", type="read"} 0 vm_tcplistener_errors_total{name="http", addr=":8429", type="write"} 0 vm_tcplistener_read_bytes_total{name="http", addr=":8429"} 36440 vm_tcplistener_read_calls_total{name="http", addr=":8429"} 120 vm_tcplistener_read_timeouts_total{name="http", addr=":8429"} 59 vm_tcplistener_write_calls_total{name="http", addr=":8429"} 1100 vm_tcplistener_write_timeouts_total{name="http", addr=":8429"} 0 vm_tcplistener_written_bytes_total{name="http", addr=":8429"} 4262338 vm_zstd_block_compress_calls_total 367 vm_zstd_block_compressed_bytes_total 20729401 vm_zstd_block_decompress_calls_total 0 vm_zstd_block_original_bytes_total 572257989 vmagent_daily_series_limit_rows_dropped_total 0 vmagent_hourly_series_limit_rows_dropped_total 0 vmagent_http_request_errors_total{path="/api/v1/import", protocol="vmimport"} 0 vmagent_http_request_errors_total{path="/api/v1/import/csv", protocol="csvimport"} 0 vmagent_http_request_errors_total{path="/api/v1/import/native", protocol="nativeimport"} 0 vmagent_http_request_errors_total{path="/api/v1/import/prometheus", protocol="prometheusimport"} 0 vmagent_http_request_errors_total{path="/api/v1/write", protocol="promremotewrite"} 0 vmagent_http_request_errors_total{path="/datadog/api/v1/series", protocol="datadog"} 0 vmagent_http_request_errors_total{path="/influx/write", protocol="influx"} 0 vmagent_http_request_errors_total{path="/target_response"} 0 vmagent_http_requests_total{path="/-/reload"} 0 vmagent_http_requests_total{path="/api/v1/import", protocol="vmimport"} 0 vmagent_http_requests_total{path="/api/v1/import/csv", protocol="csvimport"} 0 vmagent_http_requests_total{path="/api/v1/import/native", protocol="nativeimport"} 0 vmagent_http_requests_total{path="/api/v1/import/prometheus", protocol="prometheusimport"} 0 vmagent_http_requests_total{path="/api/v1/status/config"} 0 vmagent_http_requests_total{path="/api/v1/targets"} 0 vmagent_http_requests_total{path="/api/v1/write", protocol="promremotewrite"} 0 vmagent_http_requests_total{path="/config"} 0 vmagent_http_requests_total{path="/datadog/api/v1/check_run", protocol="datadog"} 0 vmagent_http_requests_total{path="/datadog/api/v1/metadata", protocol="datadog"} 0 vmagent_http_requests_total{path="/datadog/api/v1/series", protocol="datadog"} 0 vmagent_http_requests_total{path="/datadog/api/v1/validate", protocol="datadog"} 0 vmagent_http_requests_total{path="/datadog/intake", protocol="datadog"} 0 vmagent_http_requests_total{path="/influx/query", protocol="influx"} 0 vmagent_http_requests_total{path="/influx/write", protocol="influx"} 0 vmagent_http_requests_total{path="/service-discovery"} 1 vmagent_http_requests_total{path="/target_response"} 0 vmagent_http_requests_total{path="/targets"} 5 vmagent_remotewrite_block_size_bytes_bucket{vmrange="6.813e+02...7.743e+02"} 50 vmagent_remotewrite_block_size_bytes_bucket{vmrange="7.743e+02...8.799e+02"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.000e+03...1.136e+03"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.668e+03...1.896e+03"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.154e+03...2.448e+03"} 3 vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.783e+03...3.162e+03"} 3 vmagent_remotewrite_block_size_bytes_bucket{vmrange="3.594e+03...4.084e+03"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.642e+03...5.275e+03"} 2 vmagent_remotewrite_block_size_bytes_bucket{vmrange="5.275e+03...5.995e+03"} 2 vmagent_remotewrite_block_size_bytes_bucket{vmrange="5.995e+03...6.813e+03"} 3 vmagent_remotewrite_block_size_bytes_bucket{vmrange="6.813e+03...7.743e+03"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="7.743e+03...8.799e+03"} 7 vmagent_remotewrite_block_size_bytes_bucket{vmrange="8.799e+03...1.000e+04"} 4 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.000e+04...1.136e+04"} 9 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.136e+04...1.292e+04"} 10 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.292e+04...1.468e+04"} 2 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.468e+04...1.668e+04"} 16 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.668e+04...1.896e+04"} 15 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.896e+04...2.154e+04"} 17 vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.154e+04...2.448e+04"} 9 vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.448e+04...2.783e+04"} 9 vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.783e+04...3.162e+04"} 4 vmagent_remotewrite_block_size_bytes_bucket{vmrange="3.594e+04...4.084e+04"} 4 vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.084e+04...4.642e+04"} 123 vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.642e+04...5.275e+04"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="5.275e+04...5.995e+04"} 2 vmagent_remotewrite_block_size_bytes_bucket{vmrange="5.995e+04...6.813e+04"} 2 vmagent_remotewrite_block_size_bytes_bucket{vmrange="6.813e+04...7.743e+04"} 5 vmagent_remotewrite_block_size_bytes_bucket{vmrange="7.743e+04...8.799e+04"} 2 vmagent_remotewrite_block_size_bytes_bucket{vmrange="8.799e+04...1.000e+05"} 1 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.668e+05...1.896e+05"} 35 vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.896e+05...2.154e+05"} 328 vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.154e+05...2.448e+05"} 365 vmagent_remotewrite_block_size_bytes_sum 164234919 vmagent_remotewrite_block_size_bytes_count 1038 vmagent_remotewrite_block_size_rows_bucket{vmrange="5.995e+00...6.813e+00"} 51 vmagent_remotewrite_block_size_rows_bucket{vmrange="7.743e+00...8.799e+00"} 1 vmagent_remotewrite_block_size_rows_bucket{vmrange="2.154e+01...2.448e+01"} 1 vmagent_remotewrite_block_size_rows_bucket{vmrange="3.594e+01...4.084e+01"} 3 vmagent_remotewrite_block_size_rows_bucket{vmrange="4.642e+01...5.275e+01"} 3 vmagent_remotewrite_block_size_rows_bucket{vmrange="5.995e+01...6.813e+01"} 1 vmagent_remotewrite_block_size_rows_bucket{vmrange="7.743e+01...8.799e+01"} 2 vmagent_remotewrite_block_size_rows_bucket{vmrange="8.799e+01...1.000e+02"} 2 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.000e+02...1.136e+02"} 3 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.136e+02...1.292e+02"} 1 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.292e+02...1.468e+02"} 2 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.468e+02...1.668e+02"} 5 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.668e+02...1.896e+02"} 9 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.896e+02...2.154e+02"} 8 vmagent_remotewrite_block_size_rows_bucket{vmrange="2.154e+02...2.448e+02"} 7 vmagent_remotewrite_block_size_rows_bucket{vmrange="2.448e+02...2.783e+02"} 15 vmagent_remotewrite_block_size_rows_bucket{vmrange="2.783e+02...3.162e+02"} 17 vmagent_remotewrite_block_size_rows_bucket{vmrange="3.162e+02...3.594e+02"} 14 vmagent_remotewrite_block_size_rows_bucket{vmrange="3.594e+02...4.084e+02"} 10 vmagent_remotewrite_block_size_rows_bucket{vmrange="4.084e+02...4.642e+02"} 11 vmagent_remotewrite_block_size_rows_bucket{vmrange="4.642e+02...5.275e+02"} 4 vmagent_remotewrite_block_size_rows_bucket{vmrange="6.813e+02...7.743e+02"} 3 vmagent_remotewrite_block_size_rows_bucket{vmrange="7.743e+02...8.799e+02"} 125 vmagent_remotewrite_block_size_rows_bucket{vmrange="8.799e+02...1.000e+03"} 1 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.000e+03...1.136e+03"} 2 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.136e+03...1.292e+03"} 5 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.292e+03...1.468e+03"} 2 vmagent_remotewrite_block_size_rows_bucket{vmrange="1.468e+03...1.668e+03"} 2 vmagent_remotewrite_block_size_rows_bucket{vmrange="3.162e+03...3.594e+03"} 380 vmagent_remotewrite_block_size_rows_bucket{vmrange="3.594e+03...4.084e+03"} 348 vmagent_remotewrite_block_size_rows_sum 2817691 vmagent_remotewrite_block_size_rows_count 1038 vmagent_remotewrite_blocks_sent_total{url="1:secret-url"} 1038 vmagent_remotewrite_bytes_sent_total{url="1:secret-url"} 164234919 vmagent_remotewrite_conn_bytes_read_total 66368 vmagent_remotewrite_conn_bytes_written_total 164495017 vmagent_remotewrite_conn_read_errors_total 17 vmagent_remotewrite_conn_reads_total 1054 vmagent_remotewrite_conn_write_errors_total 0 vmagent_remotewrite_conn_writes_total 4594 vmagent_remotewrite_conns 5 vmagent_remotewrite_dial_errors_total 0 vmagent_remotewrite_dials_total 22 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="2.448e-01...2.783e-01"} 6 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="2.783e-01...3.162e-01"} 29 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="3.162e-01...3.594e-01"} 149 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="3.594e-01...4.084e-01"} 94 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="4.084e-01...4.642e-01"} 137 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="4.642e-01...5.275e-01"} 503 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="5.275e-01...5.995e-01"} 107 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="5.995e-01...6.813e-01"} 10 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="6.813e-01...7.743e-01"} 1 vmagent_remotewrite_duration_seconds_bucket{url="1:secret-url",vmrange="7.743e-01...8.799e-01"} 1 vmagent_remotewrite_duration_seconds_sum{url="1:secret-url"} 472.4915656079999 vmagent_remotewrite_duration_seconds_count{url="1:secret-url"} 1037 vmagent_remotewrite_errors_total{url="1:secret-url"} 0 vmagent_remotewrite_global_relabel_metrics_dropped_total 0 vmagent_remotewrite_global_rows_pushed_before_relabel_total 2821290 vmagent_remotewrite_packets_dropped_total{url="1:secret-url"} 0 vmagent_remotewrite_pending_data_bytes{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE", url="1:secret-url"} 0 vmagent_remotewrite_pending_inmemory_blocks{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE", url="1:secret-url"} 0 vmagent_remotewrite_queues{url="1:secret-url"} 420 vmagent_remotewrite_rate_limit_reached_total{url="1:secret-url"} 0 vmagent_remotewrite_rate_limit{url="1:secret-url"} 0 vmagent_remotewrite_relabel_metrics_dropped_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE", url="1:secret-url"} 0 vmagent_remotewrite_requests_total{url="1:secret-url", status_code="2XX"} 1037 vmagent_remotewrite_retries_count_total{url="1:secret-url"} 0 vmagent_remotewrite_rows_pushed_after_relabel_total{path="vmagent-remotewrite-data/persistent-queue/1_2AD36D824210E0DE", url="1:secret-url"} 2821290 vmagent_remotewrite_send_duration_seconds_total{url="1:secret-url"} 472.5182890020004 vmagent_rows_inserted_total{type="csvimport"} 0 vmagent_rows_inserted_total{type="datadog"} 0 vmagent_rows_inserted_total{type="graphite"} 0 vmagent_rows_inserted_total{type="influx"} 0 vmagent_rows_inserted_total{type="native"} 0 vmagent_rows_inserted_total{type="opentsdb"} 0 vmagent_rows_inserted_total{type="opentsdbhttp"} 0 vmagent_rows_inserted_total{type="prometheus"} 0 vmagent_rows_inserted_total{type="promremotewrite"} 0 vmagent_rows_inserted_total{type="vmimport"} 0 go_memstats_alloc_bytes 527456720 go_memstats_alloc_bytes_total 28711966688 go_memstats_buck_hash_sys_bytes 1656028 go_memstats_frees_total 2290982 go_memstats_gc_cpu_fraction 0.0011250648000318076 go_memstats_gc_sys_bytes 29694144 go_memstats_heap_alloc_bytes 527456720 go_memstats_heap_idle_bytes 108265472 go_memstats_heap_inuse_bytes 531300352 go_memstats_heap_objects 138087 go_memstats_heap_released_bytes 34725888 go_memstats_heap_sys_bytes 639565824 go_memstats_last_gc_time_seconds 1.6667137988492596e+09 go_memstats_lookups_total 0 go_memstats_mallocs_total 2429069 go_memstats_mcache_inuse_bytes 38400 go_memstats_mcache_sys_bytes 46800 go_memstats_mspan_inuse_bytes 765272 go_memstats_mspan_sys_bytes 1028160 go_memstats_next_gc_bytes 591956857 go_memstats_other_sys_bytes 4676844 go_memstats_stack_inuse_bytes 10551296 go_memstats_stack_sys_bytes 10551296 go_memstats_sys_bytes 687219096 go_cgo_calls_count 1217 go_cpu_count 16 go_gc_duration_seconds{quantile="0"} 5.1905e-05 go_gc_duration_seconds{quantile="0.25"} 6.7101e-05 go_gc_duration_seconds{quantile="0.5"} 7.5764e-05 go_gc_duration_seconds{quantile="0.75"} 8.6788e-05 go_gc_duration_seconds{quantile="1"} 0.001804597 go_gc_duration_seconds_sum 0.032223673 go_gc_duration_seconds_count 335 go_gc_forced_count 0 go_gomaxprocs 32 go_goroutines 2494 go_threads 37 go_info{version="go1.19.2"} 1 go_info_ext{compiler="gc", GOARCH="amd64", GOOS="linux", GOROOT=""} 1 process_cpu_seconds_system_total 2.67 process_cpu_seconds_total 68.09 process_cpu_seconds_user_total 65.42 process_major_pagefaults_total 0 process_minor_pagefaults_total 323181 process_num_threads 36 process_resident_memory_bytes 649707520 process_start_time_seconds 1666713083 process_virtual_memory_bytes 1376997376 process_virtual_memory_peak_bytes 1379258368 process_resident_memory_peak_bytes 678572032 process_resident_memory_anon_bytes 640589824 process_resident_memory_file_bytes 9117696 process_resident_memory_shared_bytes 0 process_io_read_bytes_total 597468269 process_io_written_bytes_total 168892943 process_io_read_syscalls_total 34791 process_io_write_syscalls_total 8910 process_io_storage_read_bytes_total 0 process_io_storage_written_bytes_total 0 process_max_fds 1048576 process_open_fds 572 vm_app_version{version="vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e", short_version=""} 1 vm_allowed_memory_bytes 10307921510 vm_available_memory_bytes 17179869184 vm_available_cpu_cores 32 vm_gogc 30 vm_app_start_timestamp 1666713083 vm_app_uptime_seconds 717 flag{name="configAuthKey", value="secret", is_set="false"} 1 flag{name="csvTrimTimestamp", value="1ms", is_set="false"} 1 flag{name="datadog.maxInsertRequestSize", value="67108864", is_set="false"} 1 flag{name="datadog.sanitizeMetricName", value="true", is_set="false"} 1 flag{name="denyQueryTracing", value="false", is_set="false"} 1 flag{name="dryRun", value="false", is_set="false"} 1 flag{name="enableTCP6", value="false", is_set="false"} 1 flag{name="envflag.enable", value="false", is_set="false"} 1 flag{name="envflag.prefix", value="", is_set="false"} 1 flag{name="flagsAuthKey", value="secret", is_set="false"} 1 flag{name="fs.disableMmap", value="false", is_set="false"} 1 flag{name="graphiteListenAddr", value="", is_set="false"} 1 flag{name="graphiteTrimTimestamp", value="1s", is_set="false"} 1 flag{name="http.connTimeout", value="2m0s", is_set="false"} 1 flag{name="http.disableResponseCompression", value="false", is_set="false"} 1 flag{name="http.idleConnTimeout", value="1m0s", is_set="false"} 1 flag{name="http.maxGracefulShutdownDuration", value="7s", is_set="false"} 1 flag{name="http.pathPrefix", value="", is_set="false"} 1 flag{name="http.shutdownDelay", value="0s", is_set="false"} 1 flag{name="httpAuth.password", value="secret", is_set="false"} 1 flag{name="httpAuth.username", value="", is_set="false"} 1 flag{name="httpListenAddr", value=":8429", is_set="false"} 1 flag{name="import.maxLineLen", value="104857600", is_set="false"} 1 flag{name="influx.databaseNames", value="", is_set="false"} 1 flag{name="influx.maxLineSize", value="262144", is_set="false"} 1 flag{name="influxDBLabel", value="db", is_set="false"} 1 flag{name="influxListenAddr", value="", is_set="false"} 1 flag{name="influxMeasurementFieldSeparator", value="_", is_set="false"} 1 flag{name="influxSkipMeasurement", value="false", is_set="false"} 1 flag{name="influxSkipSingleField", value="false", is_set="false"} 1 flag{name="influxTrimTimestamp", value="1ms", is_set="false"} 1 flag{name="insert.maxQueueDuration", value="1m0s", is_set="false"} 1 flag{name="loggerDisableTimestamps", value="false", is_set="false"} 1 flag{name="loggerErrorsPerSecondLimit", value="0", is_set="false"} 1 flag{name="loggerFormat", value="default", is_set="false"} 1 flag{name="loggerLevel", value="INFO", is_set="false"} 1 flag{name="loggerOutput", value="stderr", is_set="false"} 1 flag{name="loggerTimezone", value="UTC", is_set="false"} 1 flag{name="loggerWarnsPerSecondLimit", value="0", is_set="false"} 1 flag{name="maxConcurrentInserts", value="128", is_set="true"} 1 flag{name="maxInsertRequestSize", value="10MB", is_set="true"} 1 flag{name="memory.allowedBytes", value="0", is_set="false"} 1 flag{name="memory.allowedPercent", value="60", is_set="false"} 1 flag{name="metricsAuthKey", value="secret", is_set="false"} 1 flag{name="opentsdbHTTPListenAddr", value="", is_set="false"} 1 flag{name="opentsdbListenAddr", value="", is_set="false"} 1 flag{name="opentsdbTrimTimestamp", value="1s", is_set="false"} 1 flag{name="opentsdbhttp.maxInsertRequestSize", value="33554432", is_set="false"} 1 flag{name="opentsdbhttpTrimTimestamp", value="1ms", is_set="false"} 1 flag{name="pprofAuthKey", value="secret", is_set="false"} 1 flag{name="promscrape.azureSDCheckInterval", value="1m0s", is_set="false"} 1 flag{name="promscrape.cluster.memberNum", value="0", is_set="false"} 1 flag{name="promscrape.cluster.membersCount", value="0", is_set="false"} 1 flag{name="promscrape.cluster.name", value="", is_set="false"} 1 flag{name="promscrape.cluster.replicationFactor", value="1", is_set="false"} 1 flag{name="promscrape.config", value="/etc/prometheus/prometheus.yaml", is_set="true"} 1 flag{name="promscrape.config.dryRun", value="false", is_set="false"} 1 flag{name="promscrape.config.strictParse", value="true", is_set="false"} 1 flag{name="promscrape.configCheckInterval", value="0s", is_set="false"} 1 flag{name="promscrape.consul.waitTime", value="0s", is_set="false"} 1 flag{name="promscrape.consulSDCheckInterval", value="30s", is_set="false"} 1 flag{name="promscrape.digitaloceanSDCheckInterval", value="1m0s", is_set="false"} 1 flag{name="promscrape.disableCompression", value="false", is_set="false"} 1 flag{name="promscrape.disableKeepAlive", value="false", is_set="false"} 1 flag{name="promscrape.discovery.concurrency", value="100", is_set="false"} 1 flag{name="promscrape.discovery.concurrentWaitTime", value="1m0s", is_set="false"} 1 flag{name="promscrape.dnsSDCheckInterval", value="30s", is_set="false"} 1 flag{name="promscrape.dockerSDCheckInterval", value="30s", is_set="false"} 1 flag{name="promscrape.dockerswarmSDCheckInterval", value="30s", is_set="false"} 1 flag{name="promscrape.dropOriginalLabels", value="false", is_set="false"} 1 flag{name="promscrape.ec2SDCheckInterval", value="1m0s", is_set="false"} 1 flag{name="promscrape.eurekaSDCheckInterval", value="30s", is_set="false"} 1 flag{name="promscrape.fileSDCheckInterval", value="1m0s", is_set="false"} 1 flag{name="promscrape.gceSDCheckInterval", value="1m0s", is_set="false"} 1 flag{name="promscrape.httpSDCheckInterval", value="1m0s", is_set="false"} 1 flag{name="promscrape.kubernetes.apiServerTimeout", value="30m0s", is_set="false"} 1 flag{name="promscrape.kubernetesSDCheckInterval", value="30s", is_set="true"} 1 flag{name="promscrape.maxDroppedTargets", value="1000", is_set="false"} 1 flag{name="promscrape.maxResponseHeadersSize", value="4096", is_set="false"} 1 flag{name="promscrape.maxScrapeSize", value="16777216", is_set="false"} 1 flag{name="promscrape.minResponseSizeForStreamParse", value="1000000", is_set="false"} 1 flag{name="promscrape.noStaleMarkers", value="false", is_set="false"} 1 flag{name="promscrape.openstackSDCheckInterval", value="30s", is_set="false"} 1 flag{name="promscrape.seriesLimitPerTarget", value="0", is_set="false"} 1 flag{name="promscrape.streamParse", value="false", is_set="false"} 1 flag{name="promscrape.suppressDuplicateScrapeTargetErrors", value="false", is_set="false"} 1 flag{name="promscrape.suppressScrapeErrors", value="false", is_set="false"} 1 flag{name="promscrape.suppressScrapeErrorsDelay", value="0s", is_set="false"} 1 flag{name="promscrape.yandexcloudSDCheckInterval", value="30s", is_set="false"} 1 flag{name="pushmetrics.extraLabel", value="", is_set="false"} 1 flag{name="pushmetrics.interval", value="10s", is_set="false"} 1 flag{name="pushmetrics.url", value="secret", is_set="false"} 1 flag{name="remoteWrite.aws.accessKey", value="secret", is_set="false"} 1 flag{name="remoteWrite.aws.ec2Endpoint", value="", is_set="false"} 1 flag{name="remoteWrite.aws.region", value="", is_set="false"} 1 flag{name="remoteWrite.aws.roleARN", value="", is_set="false"} 1 flag{name="remoteWrite.aws.secretKey", value="secret", is_set="false"} 1 flag{name="remoteWrite.aws.service", value="", is_set="false"} 1 flag{name="remoteWrite.aws.stsEndpoint", value="", is_set="false"} 1 flag{name="remoteWrite.aws.useSigv4", value="", is_set="false"} 1 flag{name="remoteWrite.basicAuth.password", value="secret", is_set="false"} 1 flag{name="remoteWrite.basicAuth.passwordFile", value="secret", is_set="false"} 1 flag{name="remoteWrite.basicAuth.username", value="", is_set="false"} 1 flag{name="remoteWrite.bearerToken", value="secret", is_set="false"} 1 flag{name="remoteWrite.bearerTokenFile", value="secret", is_set="false"} 1 flag{name="remoteWrite.flushInterval", value="1s", is_set="false"} 1 flag{name="remoteWrite.headers", value="", is_set="false"} 1 flag{name="remoteWrite.label", value="", is_set="false"} 1 flag{name="remoteWrite.maxBlockSize", value="8388608", is_set="false"} 1 flag{name="remoteWrite.maxDailySeries", value="0", is_set="false"} 1 flag{name="remoteWrite.maxDiskUsagePerURL", value="", is_set="false"} 1 flag{name="remoteWrite.maxHourlySeries", value="0", is_set="false"} 1 flag{name="remoteWrite.maxRowsPerBlock", value="10000", is_set="false"} 1 flag{name="remoteWrite.multitenantURL", value="", is_set="false"} 1 flag{name="remoteWrite.oauth2.clientID", value="", is_set="false"} 1 flag{name="remoteWrite.oauth2.clientSecret", value="secret", is_set="false"} 1 flag{name="remoteWrite.oauth2.clientSecretFile", value="secret", is_set="false"} 1 flag{name="remoteWrite.oauth2.scopes", value="", is_set="false"} 1 flag{name="remoteWrite.oauth2.tokenUrl", value="secret", is_set="false"} 1 flag{name="remoteWrite.proxyURL", value="", is_set="false"} 1 flag{name="remoteWrite.queues", value="420", is_set="true"} 1 flag{name="remoteWrite.rateLimit", value="", is_set="false"} 1 flag{name="remoteWrite.relabelConfig", value="", is_set="false"} 1 flag{name="remoteWrite.relabelDebug", value="false", is_set="false"} 1 flag{name="remoteWrite.roundDigits", value="", is_set="false"} 1 flag{name="remoteWrite.sendTimeout", value="", is_set="false"} 1 flag{name="remoteWrite.showURL", value="false", is_set="false"} 1 flag{name="remoteWrite.significantFigures", value="", is_set="false"} 1 flag{name="remoteWrite.tlsCAFile", value="", is_set="false"} 1 flag{name="remoteWrite.tlsCertFile", value="", is_set="false"} 1 flag{name="remoteWrite.tlsInsecureSkipVerify", value="", is_set="false"} 1 flag{name="remoteWrite.tlsKeyFile", value="secret", is_set="false"} 1 flag{name="remoteWrite.tlsServerName", value="", is_set="false"} 1 flag{name="remoteWrite.tmpDataPath", value="vmagent-remotewrite-data", is_set="false"} 1 flag{name="remoteWrite.url", value="secret", is_set="true"} 1 flag{name="remoteWrite.urlRelabelConfig", value="", is_set="false"} 1 flag{name="remoteWrite.urlRelabelDebug", value="", is_set="false"} 1 flag{name="sortLabels", value="false", is_set="false"} 1 flag{name="tls", value="false", is_set="false"} 1 flag{name="tlsCertFile", value="", is_set="false"} 1 flag{name="tlsCipherSuites", value="", is_set="false"} 1 flag{name="tlsKeyFile", value="secret", is_set="false"} 1 flag{name="tlsMinVersion", value="", is_set="false"} 1 flag{name="usePromCompatibleNaming", value="false", is_set="false"} 1 flag{name="version", value="false", is_set="false"} 1 ```

Version

$ ./vmagent-prod --version
vmagent-20221021-205625-heads-public-single-node-0-g4128ad71e

Used command-line flags

    command:
      - /vmagent-prod
      - -promscrape.config=/etc/prometheus/prometheus.yaml
      - -remoteWrite.url=http://*********:9201
      - -maxInsertRequestSize=10MB
      - -maxConcurrentInserts=128
      - -maxInsertRequestSize=10MB

VictoriaMetrics / VictoriaMetrics

VM Agent: slow proceccing and insufficient resource utilization #3272