(sometimes) skyrocketing announce latency, drop in requests per second

[ckcr4lyf]

need to investigate cause... Announce latency goes upto 7+ seconds!

[ckcr4lyf]

"Normal" behavior - sub millisecond latency, not too many FIN_WAIT or ESTABLISHED

[Power2All]

I would suggest you look at Axum. I've ditched Actix for having also some memory leak issues. Seeing you already follow my Torrust-Axum project, you probably are maybe already aware :)

[ckcr4lyf]

Yep, need to dig deeper into whether the cheap VPS is limiting or it really is Actix

[Power2All]

Yep, need to dig deeper into whether the cheap VPS is limiting or it really is Actix

I've got my own hardware I put in a datacenter, although it runs in a VM. VM is most likely the culprit, I would suggest you monitor also the amount of active packets, and/or drops through /var/log/syslog, and you can use the command ss -s to see active TCP and UDP connections.

Most possible, you would see some info in the syslog file, just use a command like tail -f /var/log/syslog to 'watch' the file with additions ;)

[ckcr4lyf]

Triage: Seems this is due to redis sets. Currently, for any torrent, we call ZRANGEBYSCORE to get the seeders & leechers on two ZSETs:

https://github.com/ckcr4lyf/kiryuu/blob/bf6d0bad158a8c558b775e2ffd98c2b487a3d7f4/src/main.rs#L71-L74

Redis has a useful --bigkeys command, using which we can find the largest ZSET:

$ redis-cli --bigkeys --pattern '*_seeders'

[...truncated]

Sampled 427012 keys in the keyspace!
Total key length in bytes is 20496576 (avg len 48.00)

Biggest   zset found '"17d0442b088a570dd94f98c6b367b98494704171_seeders"' has 30452 members

This torrent has ~30k seeders. As a result, whenever any of those 30k seeders (or ~10k leechers I also discovered) announces, kiryuu calls ZRANGEBYSCORE on two ZSETs with ~30k & ~10k members.

Additionally the leechers are also shuffled, which can be CPU expensive: https://github.com/ckcr4lyf/kiryuu/blob/bf6d0bad158a8c558b775e2ffd98c2b487a3d7f4/src/main.rs#L86

[ckcr4lyf]

This is most evident when we see stats for mywaifu. Notice how the average number of commands redis executes stays the same throughout the ~30hr time period.

However, ~2022-01-17T19:00:00 onwards, the HTTP announce latency skyrockets. The redis netwok I/O also increases massively, and most time is spent by the ZRANGEBYSCORE command. This is most likely due to the massive number of peers on the torrent, which are very expensive on the CPU (executing ZRANGEBYSCORE on a ZSET with 30k+ members by Redis, as well as shuffling within kiryuu probably)

[ckcr4lyf]

Possible Soutions

Do not return all elements in ZRANGEBYSCORE

This will reduce network traffic & CPU load on redis. The time complexity is "O(log(N)+M) with N being the number of elements in the sorted set and M the number of elements returned." (Ref: https://redis.io/commands/zrange/)

Instead , we can get the last N elements of the ZSET, and then shuffle those.

Pros:

• Less CPU load on redis (ZRANGEBYSCORE)
• Less CPU load on kiryuu (rand shuffle)
• Less network traffic between kiryuu & redis (no need to transfer 30k peers everytime)

Cons:

• The N+1th element would almost never be available to the peer whose announce got the first N (Explained below).
• We cannot determine if the announcer is a seeder or leecher from the response of ZRANGEBYSCORE (since it is incomplete). Instead we would need to make another redis call, probably ZSCORE (as it is O(1)).

Explaining the N+1th scenario

BitTorrent works by announcing at fixed intervals, usually every 30 min.

If Peer A announces at t = 0, And then N peers announce at 5 <= t <= 25 , and then Peer B announces at t=29, then Peer B will be limited to N peers, and would not see Peer A. This isn't a massive downside, especially with DHT & such aiding in peer discovery once you have a few peers.

[ckcr4lyf]

FYI: COmparing the CPU load (on VPS) with redis queries... gg...

[ckcr4lyf]

This has been fixed by:

• Adding a cache before hitting redis with ZRANGE (https://github.com/ckcr4lyf/kiryuu/pull/14)
• Checking if announcer is seeder/leecher via ZSCORE (rather than traversing the vector) (https://github.com/ckcr4lyf/kiryuu/pull/14)
• Limiting ZRANGE to 50 entries (https://github.com/ckcr4lyf/kiryuu/pull/20 , https://github.com/ckcr4lyf/kiryuu/issues/19)

Some other micro optimizations include:

• Improving how we parse the query into a String: (https://github.com/ckcr4lyf/kiryuu/pull/22 , https://github.com/ckcr4lyf/kiryuu/pull/23)
• IP parsing from Vec<u8> to [u8; 6] (https://github.com/ckcr4lyf/kiryuu/pull/24)
• Using underlying SockAddr for IP (no need to parse "1.1.1.1" from string) (https://github.com/ckcr4lyf/kiryuu/pull/26)
• Change query parsing from String to [u8; 40] , and redis keys to use [u8; _] instead of allocating String s . (https://github.com/ckcr4lyf/kiryuu/pull/27)

[ckcr4lyf]

Overall result: Can now process ~2500 announce per second while maintaining sub millisecond latency

2 week view: