search

maxisam / mgob

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.
https://maxisam.github.io/mgob/
MIT License
143 stars 21 forks source link

Missing metrics #133

Open reamasesa opened 5 days ago

reamasesa commented 5 days ago

Hi,

I'm missing the backup metrics from mgob even tough my backup succeed

curl http://localhost:8090/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.0806e-05
go_gc_duration_seconds{quantile="0.25"} 5.0806e-05
go_gc_duration_seconds{quantile="0.5"} 5.0806e-05
go_gc_duration_seconds{quantile="0.75"} 5.0806e-05
go_gc_duration_seconds{quantile="1"} 5.0806e-05
go_gc_duration_seconds_sum 5.0806e-05
go_gc_duration_seconds_count 1
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 10
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.21.5"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.712728e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.514784e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.447208e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 9512
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.324032e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.712728e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.629632e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.775936e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 4387
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 2.228224e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.405568e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7295915429722137e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 13899
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 19200
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 31200
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 121800
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 130368
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.757248e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 950272
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 950272
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.5045896e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 13
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.03
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.8481152e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.72959153994e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.274417152e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 2
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

From the backup log file

2024-10-22T09:53:26.296+0000    writing tester.Tester to archive '/tmp/tester-1729590806.gz'
2024-10-22T09:53:26.296+0000    done dumping tester.meetings-report-queue (0 documents)
2024-10-22T09:53:26.296+0000    done dumping tester.potential-meetings-reports (0 documents)
2024-10-22T09:53:26.296+0000    done dumping tester.reports (1 document)
2024-10-22T09:53:26.296+0000    writing tester.identifiers to archive '/tmp/tester-1729590806.gz'
2024-10-22T09:53:26.296+0000    writing tester.credits to archive '/tmp/tester-1729590806.gz'
2024-10-22T09:53:26.297+0000    done dumping tester.Tester (0 documents)
2024-10-22T09:53:26.298+0000    done dumping tester.credits (0 documents)
2024-10-22T09:53:26.298+0000    done dumping tester.identifiers (0 documents)

From the container's log

mgob  | time="2024-10-22T09:53:26Z" level=info msg="On demand backup started" plan=tester
mgob  | time="2024-10-22T09:53:26Z" level=debug msg="dump cmd: mongodump --archive=/tmp/tester-1729590806.gz --gzip --host mongo --port 27017 -u \"${MONGODB_ADMIN_USERNAME}\" -p xxxx --db tester --authenticationDatabase admin " plan=tester
mgob  | time="2024-10-22T09:53:26Z" level=info msg="new dump" archive=/tmp/tester-1729590806.gz err="<nil>" mlog=/tmp/tester-1729590806.log plan=tester
mgob  | time="2024-10-22T09:53:26Z" level=debug msg="applying retention to *.gz* files"
mgob  | time="2024-10-22T09:53:26Z" level=debug msg="applying retention to *.log files"
mgob  | time="2024-10-22T09:53:26Z" level=info msg="Local backup finished filename:`/tmp/tester-1729590806.gz`, filepath:`/storage/tester/tester-1729590806.gz`, Duration: 2.145033ms" plan=tester
mgob  | time="2024-10-22T09:53:26Z" level=info msg="S3 upload finished `/tmp/tester-1729590806.gz` -> `tester/tester-backups/tester/tester-1729590806.gz` Total: 151.76 KiB, Transferred: 151.76 KiB, Speed: 1.09 MiB/s " plan=tester
mgob  | time="2024-10-22T09:53:26Z" level=info msg="Clean up temp finished Temp folder cleanup finished, `/tmp/tester-1729590806.gz` is removed." plan=tester
mgob  | time="2024-10-22T09:53:26Z" level=info msg="On demand backup finished in 302.806218ms archive tester-1729590806.gz size 155 kB" plan=tester

Excerpt from the docker-compose.yml file

  mgob:
    <<: *service-defaults
    image: "maxisam/mgob:${MGOB_VERSION}"
    container_name: 'mgob'
    depends_on:
      # Don't start before mongo is up and running.
      mongo:
        condition: 'service_healthy'
    ports:
      - "${MGOB_PORT}:${MGOB_PORT}"
    command:
      - '--LogLevel=debug'
    environment:
      MONGODB_ADMIN_USERNAME: "${MONGODB_ADMIN_USERNAME}"
      MONGODB_ADMIN_PASSWORD: "${MONGODB_ADMIN_PASSWORD}"
      MGOB_BUCKET_NAME_AND_PATH: "${MGOB_BUCKET_NAME_AND_PATH}"
    volumes:
      - "${MGOB_CONFIG}:/config"
      - "${MGOB_DATA}:/data"
      - "${MGOB_STORAGE}:/storage"
      - "${MGOB_TMP}:/tmp"

mgob version: 2.0.24

If any information is missing or required, let me know, I'll collect what I can

Thanks

maxisam commented 3 days ago

I think it only works for schedule backup. If you trigger it from api, i don't think it will work

boomfly commented 16 hours ago

@maxisam you right, scheduler store mgob metrics scheduler.go: 116