Background

IPFS is a peer-to-peer (p2p) storage network. Content is accessible through peers that might relay information or store it (or do both), and those peers can be located anywhere in the world. IPFS knows how to find what you ask for by its content address, rather than where it is.

source: https://docs.ipfs.io/introduction/how-ipfs-works/

Simply running ipfs daemon will turn on your IPFS node, and will be running in your terminal window as long as it is open and your connection to the server is maintained.

source: https://medium.com/@rossbulat/introduction-to-ipfs-set-up-nodes-on-your-network-with-http-gateways-10e21ea689a4

Github Stats

https://github.com/ipfs/go-ipfs (9k stars) https://github.com/ipfs/ipfs (17k stars)

Scraping

HTTP API: https://docs.ipfs.io/reference/api/http/ 1.1 Logs tail: events 1.2 Repo/FS/Peer stats 1.2 Prometheus metrics 1.3 /debug/pprof , /debug/vars, etc.

Sample metrics

Log tail

{"event":"dhtSentMessage","message":{"key":"QmaD6C5uyXPSgPoRKTStg7FmBuxbjvGwk7pNHCtLsvpnos","type":"FIND_NODE"},"peerID":"12D3KooWM6s7X37oAFSVGH3EZyXH1QXsFgWKQnY5NrW4vribXjcF","system":"dht","time":"2019-12-10T13:05:19.3656958Z"}
{"event":"updatePeer","peerID":"12D3KooWM6s7X37oAFSVGH3EZyXH1QXsFgWKQnY5NrW4vribXjcF","system":"dht","time":"2019-12-10T13:05:19.3658334Z"}
{"event":"dhtReceivedMessage","message":{"key":"","type":"FIND_NODE"},"peerID":"12D3KooWM6s7X37oAFSVGH3EZyXH1QXsFgWKQnY5NrW4vribXjcF","system":"dht","time":"2019-12-10T13:05:19.365892Z"}
{"event":"dhtSentMessage","message":{"key":"QmXgqKTbzdh83pQtKFb19SpMCpDDcKR2ujqk3pKph9aCNF","type":"FIND_NODE"},"peerID":"QmZsCtbN19j7CTN7WsTUxt4mAeTenRsRMARH6U2CdARQHS","system":"dht","time":"2019-12-10T13:05:19.3855198Z"}
{"event":"updatePeer","peerID":"QmZsCtbN19j7CTN7WsTUxt4mAeTenRsRMARH6U2CdARQHS","system":"dht","time":"2019-12-10T13:05:19.3856654Z"}
{"event":"dhtReceivedMessage","message":{"key":"","type":"FIND_NODE"},"peerID":"QmZsCtbN19j7CTN7WsTUxt4mAeTenRsRMARH6U2CdARQHS","system":"dht","time":"2019-12-10T13:05:19.3857314Z"}
{"event":"dhtSentMessage","message":{"key":"QmaD6C5uyXPSgPoRKTStg7FmBuxbjvGwk7pNHCtLsvpnos","type":"FIND_NODE"},"peerID":"QmQmfPz9Xn4cNE6vfWcfrozeNDCx9BJFCdRMM3Cnmx2226","system":"dht","time":"2019-12-10T13:05:19.4276136Z"}
{"event":"updatePeer","peerID":"QmQmfPz9Xn4cNE6vfWcfrozeNDCx9BJFCdRMM3Cnmx2226","system":"dht","time":"2019-12-10T13:05:19.4276864Z"}
{"event":"dhtReceivedMessage","message":{"key":"","type":"FIND_NODE"},"peerID":"QmQmfPz9Xn4cNE6vfWcfrozeNDCx9BJFCdRMM3Cnmx2226","system":"dht","time":"2019-12-10T13:05:19.4277353Z"}
{"event":"swarmDialBackoffAdd","localPeer":"QmQLN2dnAP7TW9q4FETXzhXUjQwSaJKLTofKbTu8qbz26r","remotePeer":"QmZLGNT5CjHDujVbzSoUvjfTdKRJnMFwqipS8Ei9F33QR5","subsystem":"swarm","system":"swarm2","time":"2019-12-10T13:05:19.669485Z"}
{"addresses":["/p2p-circuit","/ip4/127.0.0.1/tcp/4001","/ip4/172.17.0.2/tcp/4001"],"event":"interfaceListenAddresses","system":"addrutil","time":"2019-12-10T13:05:19.6700643Z"}
{"event":"updatePeer","peerID":"QmfK9DboGJVXbYaJCuzWBrY2cnquAEfhQC6W9cj3CdAN1W","session":"cea2541c-61fd-466d-bf38-447b50b7e22c","system":"dht","time":"2019-12-10T13:05:19.7030792Z"}
{"addresses":["/p2p-circuit","/ip4/127.0.0.1/tcp/4001","/ip4/172.17.0.2/tcp/4001"],"event":"interfaceListenAddresses","system":"addrutil","time":"2019-12-10T13:05:19.7039284Z"}
{"event":"swarmDialBackoffAdd","localPeer":"QmQLN2dnAP7TW9q4FETXzhXUjQwSaJKLTofKbTu8qbz26r","remotePeer":"QmWP8ygPVRz4f4RKKhMVxQFRZKtoSW4Nq6MjUxnhn3EVC1","subsystem":"swarm","system":"swarm2","time":"2019-12-10T13:05:19.7920778Z"}
{"addresses":["/p2p-circuit","/ip4/127.0.0.1/tcp/4001","/ip4/172.17.0.2/tcp/4001"],"event":"interfaceListenAddresses","system":"addrutil","time":"2019-12-10T13:05:19.7926051Z"}
{"event":"swarmDialBackoffAdd","localPeer":"QmQLN2dnAP7TW9q4FETXzhXUjQwSaJKLTofKbTu8qbz26r","remotePeer":"QmUTzPWYcMFjKYGrzrrb91skcMCZ3wZRDQZyyZozTpdfbf","subsystem":"swarm","system":"swarm2","time":"2019-12-10T13:05:19.8207016Z"}
{"addresses":["/p2p-circuit","/ip4/127.0.0.1/tcp/4001","/ip4/172.17.0.2/tcp/4001"],"event":"interfaceListenAddresses","system":"addrutil","time":"2019-12-10T13:05:19.8213085Z"}
{"TraceID":60957560759996136,"SpanID":905828718400790363,"ParentSpanID":0,"Operation":"swarmDialDo","Start":"2019-12-10T13:05:19.8209631Z","Duration":389500,"Tags":{"system":"swarm2"},"Logs":[{"Timestamp":"2019-12-10T13:05:19.8209761Z","Fields":[{"Key":"subsystem","Value":"swarm"}]},{"Timestamp":"2019-12-10T13:05:19.8209853Z","Fields":[{"Key":"localPeer","Value":"QmQLN2dnAP7TW9q4FETXzhXUjQwSaJKLTofKbTu8qbz26r"}]},{"Timestamp":"2019-12-10T13:05:19.8209947Z","Fields":[{"Key":"remotePeer","Value":"QmSsA2wLB4tXHWjpSsCxn6udQ6a97xSCUs7fXV9e1ybNFk"}]}]}

Repo stats - /api/v0/repo/stat

{"RepoSize":3816062,"StorageMax":10000000000,"NumObjects":158,"RepoPath":"/data/ipfs","Version":"fs-repo@7"}

Prometheus metrics:

flatfs_datastore_scrub_latency_seconds_bucket{le="0.1"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="1"} 0
flatfs_datastore_scrub_latency_seconds_bucket{le="+Inf"} 0
flatfs_datastore_scrub_latency_seconds_sum 0
flatfs_datastore_scrub_latency_seconds_count 0
# HELP flatfs_datastore_scrub_total Total number of Datastore.Scrub calls
# TYPE flatfs_datastore_scrub_total counter
flatfs_datastore_scrub_total 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.35e-05
go_gc_duration_seconds{quantile="0.25"} 5.75e-05
go_gc_duration_seconds{quantile="0.5"} 0.000126
go_gc_duration_seconds{quantile="0.75"} 0.000214
go_gc_duration_seconds{quantile="1"} 0.0067905
go_gc_duration_seconds_sum 0.021175
go_gc_duration_seconds_count 82
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 993
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.7"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.7390984e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.176082712e+09
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.652042e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter

Metricbeat Module / Dataset release checklist

This checklist is intended for Devs which create or update a module to make sure modules are consistent.

Modules

For a metricset to go GA, the following criterias should be met:

[ ] Supported versions are documented
[ ] Supported operating systems are documented (if applicable)
[ ] Integration tests exist
[ ] System tests exist
[ ] Automated checks that all fields are documented
[ ] Documentation
[ ] Fields follow ECS and naming conventions
[ ] Dashboards exists (if applicable)
[ ] Kibana Home Tutorial (if applicable)
- Open PR against Kibana repo with tutorial. Examples can be found here.

Filebeat module

[ ] Test log files exist for the grok patterns
[ ] Generated output for at least 1 log file exists

Metricbeat module

[ ] Example data.json exists and an automated way to generate it exists (go test -data)
[ ] Test environment in Docker exist for integration tests

elastic / beats

[Metricbeat] IPFS Daemon Module #15024