dotnet-monitor captures incomplete metrics

zhenlei520 commented 2 months ago

After installing dotnet-monitor globally, some indicators are missing (microsoftaspnetcorehosting_current_requests, microsoftaspnetcorehosting_failed_requests)

dotnet tool install -g dotnet-monitor

Two machines, one of which is normal, and the other is wrong, but I tried to reinstall dotnet-monitor and it didn't work

/metrics

# HELP systemruntime_cpu_usage_ratio CPU Usage
# TYPE systemruntime_cpu_usage_ratio gauge
systemruntime_cpu_usage_ratio 0.7682154380157972 1724232647840
systemruntime_cpu_usage_ratio 2.1457072681275133 1724232652840
systemruntime_cpu_usage_ratio 1.0855099005159785 1724232657840
# HELP systemruntime_working_set_bytes Working Set
# TYPE systemruntime_working_set_bytes gauge
systemruntime_working_set_bytes 378912768 1724232647840
systemruntime_working_set_bytes 379146240 1724232652840
systemruntime_working_set_bytes 379240448 1724232657840
# HELP systemruntime_gc_heap_size_bytes GC Heap Size
# TYPE systemruntime_gc_heap_size_bytes gauge
systemruntime_gc_heap_size_bytes 39327528 1724232647840
systemruntime_gc_heap_size_bytes 46830384 1724232652840
systemruntime_gc_heap_size_bytes 49979968 1724232657840
# HELP systemruntime_gen_0_gc_count Gen 0 GC Count
# TYPE systemruntime_gen_0_gc_count gauge
systemruntime_gen_0_gc_count 0 1724232647840
systemruntime_gen_0_gc_count 0 1724232652841
systemruntime_gen_0_gc_count 0 1724232657840
# HELP systemruntime_gen_1_gc_count Gen 1 GC Count
# TYPE systemruntime_gen_1_gc_count gauge
systemruntime_gen_1_gc_count 0 1724232647840
systemruntime_gen_1_gc_count 0 1724232652841
systemruntime_gen_1_gc_count 0 1724232657840
# HELP systemruntime_gen_2_gc_count Gen 2 GC Count
# TYPE systemruntime_gen_2_gc_count gauge
systemruntime_gen_2_gc_count 0 1724232647840
systemruntime_gen_2_gc_count 0 1724232652841
systemruntime_gen_2_gc_count 0 1724232657840
# HELP systemruntime_gen_0_gc_budget_bytes Gen 0 GC Budget
# TYPE systemruntime_gen_0_gc_budget_bytes gauge
systemruntime_gen_0_gc_budget_bytes 62000000 1724232647840
systemruntime_gen_0_gc_budget_bytes 62000000 1724232652841
systemruntime_gen_0_gc_budget_bytes 62000000 1724232657840
# HELP systemruntime_threadpool_thread_count ThreadPool Thread Count
# TYPE systemruntime_threadpool_thread_count gauge
systemruntime_threadpool_thread_count 5 1724232647840
systemruntime_threadpool_thread_count 5 1724232652841
systemruntime_threadpool_thread_count 4 1724232657840
# HELP systemruntime_monitor_lock_contention_count Monitor Lock Contention Count
# TYPE systemruntime_monitor_lock_contention_count gauge
systemruntime_monitor_lock_contention_count 68 1724232647840
systemruntime_monitor_lock_contention_count 154 1724232652841
systemruntime_monitor_lock_contention_count 82 1724232657840
# HELP systemruntime_threadpool_queue_length ThreadPool Queue Length
# TYPE systemruntime_threadpool_queue_length gauge
systemruntime_threadpool_queue_length 0 1724232647840
systemruntime_threadpool_queue_length 0 1724232652841
systemruntime_threadpool_queue_length 0 1724232657840
# HELP systemruntime_threadpool_completed_items_count ThreadPool Completed Work Item Count
# TYPE systemruntime_threadpool_completed_items_count gauge
systemruntime_threadpool_completed_items_count 526 1724232647840
systemruntime_threadpool_completed_items_count 1390 1724232652841
systemruntime_threadpool_completed_items_count 874 1724232657840
# HELP systemruntime_alloc_rate_bytes Allocation Rate
# TYPE systemruntime_alloc_rate_bytes gauge
systemruntime_alloc_rate_bytes 1368784 1724232647840
systemruntime_alloc_rate_bytes 7494752 1724232652841
systemruntime_alloc_rate_bytes 3143040 1724232657840
# HELP systemruntime_active_timer_count Number of Active Timers
# TYPE systemruntime_active_timer_count gauge
systemruntime_active_timer_count 76 1724232647840
systemruntime_active_timer_count 78 1724232652841
systemruntime_active_timer_count 78 1724232657840
# HELP systemruntime_gc_fragmentation_ratio GC Fragmentation
# TYPE systemruntime_gc_fragmentation_ratio gauge
systemruntime_gc_fragmentation_ratio 53.1712099869075 1724232647840
systemruntime_gc_fragmentation_ratio 53.1712099869075 1724232652841
systemruntime_gc_fragmentation_ratio 53.1712099869075 1724232657840
# HELP systemruntime_gc_committed_bytes GC Committed Bytes
# TYPE systemruntime_gc_committed_bytes gauge
systemruntime_gc_committed_bytes 122068992 1724232647840
systemruntime_gc_committed_bytes 122068992 1724232652841
systemruntime_gc_committed_bytes 122068992 1724232657840
# HELP systemruntime_exception_count Exception Count
# TYPE systemruntime_exception_count gauge
systemruntime_exception_count 0 1724232647840
systemruntime_exception_count 0 1724232652841
systemruntime_exception_count 38 1724232657840
# HELP systemruntime_time_in_gc_ratio % Time in GC since last GC
# TYPE systemruntime_time_in_gc_ratio gauge
systemruntime_time_in_gc_ratio 0 1724232647840
systemruntime_time_in_gc_ratio 0 1724232652841
systemruntime_time_in_gc_ratio 0 1724232657840
# HELP systemruntime_total_pause_time_by_gc_ms Time paused by GC
# TYPE systemruntime_total_pause_time_by_gc_ms gauge
systemruntime_total_pause_time_by_gc_ms 0 1724232647840
systemruntime_total_pause_time_by_gc_ms 0 1724232652841
systemruntime_total_pause_time_by_gc_ms 0 1724232657840
# HELP systemruntime_gen_0_size_bytes Gen 0 Size
# TYPE systemruntime_gen_0_size_bytes gauge
systemruntime_gen_0_size_bytes 30321792 1724232647840
systemruntime_gen_0_size_bytes 30321792 1724232652841
systemruntime_gen_0_size_bytes 30321792 1724232657840
# HELP systemruntime_gen_1_size_bytes Gen 1 Size
# TYPE systemruntime_gen_1_size_bytes gauge
systemruntime_gen_1_size_bytes 2848616 1724232647840
systemruntime_gen_1_size_bytes 2848616 1724232652841
systemruntime_gen_1_size_bytes 2848616 1724232657840
# HELP systemruntime_gen_2_size_bytes Gen 2 Size
# TYPE systemruntime_gen_2_size_bytes gauge
systemruntime_gen_2_size_bytes 8361360 1724232647840
systemruntime_gen_2_size_bytes 8361360 1724232652841
systemruntime_gen_2_size_bytes 8361360 1724232657840
# HELP systemruntime_loh_size_bytes LOH Size
# TYPE systemruntime_loh_size_bytes gauge
systemruntime_loh_size_bytes 37241624 1724232647840
systemruntime_loh_size_bytes 37241624 1724232652841
systemruntime_loh_size_bytes 37241624 1724232657840
# HELP systemruntime_poh_size_bytes POH (Pinned Object Heap) Size
# TYPE systemruntime_poh_size_bytes gauge
systemruntime_poh_size_bytes 429184 1724232647840
systemruntime_poh_size_bytes 429184 1724232652841
systemruntime_poh_size_bytes 429184 1724232657840
# HELP systemruntime_assembly_count Number of Assemblies Loaded
# TYPE systemruntime_assembly_count gauge
systemruntime_assembly_count 280 1724232647840
systemruntime_assembly_count 280 1724232652841
systemruntime_assembly_count 280 1724232657840
# HELP systemruntime_il_bytes_jitted_bytes IL Bytes Jitted
# TYPE systemruntime_il_bytes_jitted_bytes gauge
systemruntime_il_bytes_jitted_bytes 2642298 1724232647840
systemruntime_il_bytes_jitted_bytes 2642334 1724232652841
systemruntime_il_bytes_jitted_bytes 2642513 1724232657840
# HELP systemruntime_methods_jitted_count Number of Methods Jitted
# TYPE systemruntime_methods_jitted_count gauge
systemruntime_methods_jitted_count 32853 1724232647840
systemruntime_methods_jitted_count 32855 1724232652841
systemruntime_methods_jitted_count 32862 1724232657840
# HELP systemruntime_time_in_jit_ms Time spent in JIT
# TYPE systemruntime_time_in_jit_ms gauge
systemruntime_time_in_jit_ms 0 1724232647840
systemruntime_time_in_jit_ms 0.43880000000081054 1724232652841
systemruntime_time_in_jit_ms 0.7570999999998094 1724232657840
# HELP microsoftaspnetcorehosting_requests_per_second Request Rate
# TYPE microsoftaspnetcorehosting_requests_per_second gauge
microsoftaspnetcorehosting_requests_per_second 26 1724232647840
microsoftaspnetcorehosting_requests_per_second 113 1724232652841
microsoftaspnetcorehosting_requests_per_second 44 1724232657841

github-actions[bot] commented 2 months ago

Welcome to dotnet-monitor!

Thanks for creating your first issue; let us know what you think of dotnet-monitor by filling out our survey.

zhenlei520 commented 2 months ago

version: 8.0.4 system: Ubuntu 22.04

wiktork commented 1 month ago

Would you be able to try collection using dotnet-counters? This will help isolate the issue.

Not necessarily related, but we have seen scenarios where there is interference between collectors (for example OpenTelemetry) or strange behavior for trimmed/self-contained apps. See https://github.com/dotnet/dotnet-monitor/issues/1241

zhenlei520 commented 1 month ago

Would you be able to try collection using dotnet-counters? This will help isolate the issue.

Not necessarily related, but we have seen scenarios where there is interference between collectors (for example OpenTelemetry) or strange behavior for trimmed/self-contained apps. See #1241

Thank you very much, I'll try it out when I have time.

piotrkryger commented 1 month ago

I have similar issue - the http.server.request.duration is not collected from .net 8.0 process. I tried with dotnet-monitor version 8.0.4+98fb44aafcfe2c2c2d11c7b4cb0c4315cdaa4510, configuration file:

{
    "$schema": "https://aka.ms/dotnet-monitor-schema",
    "DefaultProcess": {
        "Filters": [
            {
                "Key": "ProcessId",
                "Value": "<pid>",
                "MatchType": "Exact"
            }
        ]
    },
    "Metrics": {
        "Providers": [
            {
                "ProviderName": "Microsoft.AspNetCore.Hosting"
            }
        ]
    }
}

Command line:

dotnet-monitor collect --no-auth --configuration-file-path test-config.json

Other metrics from same provider are collected properly (failed_requests, current_requests, etc.).

At the same time I'm able to show this metric using dotnet-counter tool, version 8.0.532401+d98406ff95be0a8398d4c403056c5705389665f3 with command:

dotnet-counters monitor -p [pid] Microsoft.AspNetCore.Hosting

dotnet / dotnet-monitor

dotnet-monitor captures incomplete metrics #7175

Welcome to dotnet-monitor!