search

bitmagnet-io / bitmagnet

A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
https://bitmagnet.io/
MIT License
2.07k stars 80 forks source link

Stalled Out #263

Closed jerodg closed 1 month ago

jerodg commented 1 month ago

Describe the bug

It's been running for a week now and I'm stuck at 2206 items. Restarting doesn't seem to do anything either. Also, the /import endpoint doesn't work; just get "404 page not found"

To Reproduce

Steps to reproduce the behavior: Use normally

Expected behavior

Continue to crawl dht

Environment Information (Required)

Additional context

N/A

mgdigital commented 1 month ago

I'm stuck at 2206 items

This is usually a networking issue and unlikely to be a problem with the software. Can you check the metrics mentioned here and if they're increasing over time? If not then you have a networking issue that will be difficult to debug here - maybe you're behind a firewall or have some misconfiguration: https://bitmagnet.io/faq.html#ive-started-bitmagnet-for-the-first-time-and-am-not-seeing-torrents-right-away-is-something-wrong

the /import endpoint doesn't work; just get "404 page not found"

The import endpoint will not respond to GET requests, please refer to the import tutorial: https://bitmagnet.io/tutorials/import.html

jerodg commented 1 month ago

I don't have "bitmagnet_dht_crawler_persisted_total" on my /metrics endpoint.

# HELP bitmagnet_dht_ktable_hashes_added Total number of hashes added to routing table.
# TYPE bitmagnet_dht_ktable_hashes_added counter
bitmagnet_dht_ktable_hashes_added 0
# HELP bitmagnet_dht_ktable_hashes_count Number of hashes in routing table.
# TYPE bitmagnet_dht_ktable_hashes_count gauge
bitmagnet_dht_ktable_hashes_count 0
# HELP bitmagnet_dht_ktable_hashes_dropped Total number of hashes dropped from routing table.
# TYPE bitmagnet_dht_ktable_hashes_dropped counter
bitmagnet_dht_ktable_hashes_dropped 0
# HELP bitmagnet_dht_ktable_nodes_added Total number of nodes added to routing table.
# TYPE bitmagnet_dht_ktable_nodes_added counter
bitmagnet_dht_ktable_nodes_added 0
# HELP bitmagnet_dht_ktable_nodes_count Number of nodes in routing table.
# TYPE bitmagnet_dht_ktable_nodes_count gauge
bitmagnet_dht_ktable_nodes_count 0
# HELP bitmagnet_dht_ktable_nodes_dropped Total number of nodes dropped from routing table.
# TYPE bitmagnet_dht_ktable_nodes_dropped counter
bitmagnet_dht_ktable_nodes_dropped 0
# HELP bitmagnet_dht_server_query_concurrency Number of concurrent DHT queries.
# TYPE bitmagnet_dht_server_query_concurrency gauge
bitmagnet_dht_server_query_concurrency{query="ping"} 0
# HELP bitmagnet_dht_server_query_error_total A counter of failed DHT queries.
# TYPE bitmagnet_dht_server_query_error_total counter
bitmagnet_dht_server_query_error_total{query="ping"} 320
# HELP bitmagnet_meta_info_requester_concurrency Number of concurrent meta info requests.
# TYPE bitmagnet_meta_info_requester_concurrency gauge
bitmagnet_meta_info_requester_concurrency 0
# HELP bitmagnet_meta_info_requester_duration_seconds Duration of successful meta info requests in seconds.
# TYPE bitmagnet_meta_info_requester_duration_seconds histogram
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.005"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.01"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.025"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.05"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.1"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.25"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="0.5"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="1"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="2.5"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="5"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="10"} 0
bitmagnet_meta_info_requester_duration_seconds_bucket{le="+Inf"} 0
bitmagnet_meta_info_requester_duration_seconds_sum 0
bitmagnet_meta_info_requester_duration_seconds_count 0
# HELP bitmagnet_meta_info_requester_error_total Total number of failed meta info requests.
# TYPE bitmagnet_meta_info_requester_error_total counter
bitmagnet_meta_info_requester_error_total 0
# HELP bitmagnet_meta_info_requester_success_total Total number of successful meta info requests.
# TYPE bitmagnet_meta_info_requester_success_total counter
bitmagnet_meta_info_requester_success_total 0
# HELP bitmagnet_process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE bitmagnet_process_cpu_seconds_total counter
bitmagnet_process_cpu_seconds_total 463.36
# HELP bitmagnet_process_max_fds Maximum number of open file descriptors.
# TYPE bitmagnet_process_max_fds gauge
bitmagnet_process_max_fds 1.048576e+06
# HELP bitmagnet_process_open_fds Number of open file descriptors.
# TYPE bitmagnet_process_open_fds gauge
bitmagnet_process_open_fds 25
# HELP bitmagnet_process_resident_memory_bytes Resident memory size in bytes.
# TYPE bitmagnet_process_resident_memory_bytes gauge
bitmagnet_process_resident_memory_bytes 6.510592e+07
# HELP bitmagnet_process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE bitmagnet_process_start_time_seconds gauge
bitmagnet_process_start_time_seconds 1.71647859501e+09
# HELP bitmagnet_process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE bitmagnet_process_virtual_memory_bytes gauge
bitmagnet_process_virtual_memory_bytes 1.301037056e+09
# HELP bitmagnet_process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE bitmagnet_process_virtual_memory_max_bytes gauge
bitmagnet_process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.9602e-05
go_gc_duration_seconds{quantile="0.25"} 7.3647e-05
go_gc_duration_seconds{quantile="0.5"} 0.00011193
go_gc_duration_seconds{quantile="0.75"} 0.000200306
go_gc_duration_seconds{quantile="1"} 0.000705165
go_gc_duration_seconds_sum 0.029438584
go_gc_duration_seconds_count 199
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 44
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.22.2"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.395456e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.4574288e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.518951e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.017385e+06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.533584e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.395456e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.399104e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.1487616e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 67939
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.964928e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 2.588672e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7165019980906188e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.085324e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 28800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 31200
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 496640
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 505920
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 2.4393224e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 4.459121e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 3.473408e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 3.473408e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 3.9408904e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 29
mgdigital commented 1 month ago

Hi, you definitely have a networking issue preventing you connecting to DHT. It could be many things and so it's difficult to debug here, but it's definitely not a problem with the software. Perhaps you are running behind a firewall or you have a VPN misconfiguration. I'd suggest trying to set up from scratch on a different network/machine.