Compression

[eklitzke]

I would like to propose a mechanism for enabling the data stream between clients and servers to be compressed using zlib. Zlib is described in RFC 1950, is widely supported on many architectures, achieves good compression ratio with low CPU overhead on real-world text data, and has many implementations. I am borrowing this idea from Telnet where it is called MCCP; you can find the full details on Telnet MCCP here: https://mudhalla.net/tintin/protocols/mccp/

The way this typically works in Telnet is the server says "I support the MCCP2 option", then a client that wishes to enable compression will say "OK I want to do MCCP2", and then the server says "OK MCCP2 compression is starting now". After that point the data stream that the server sends to the client is treated as a zlib stream with an empty initial dictionary. The zlib compression only applies to the data stream of data sent by the server to the client, the data sent by the client to the server will not be compressed. Typically the stream will remain zlib compressed from this point until the connection is closed, but it is possible to end zlib compression and return to uncompressed data; the server can do this with an orderly zlib stream finish (Z_FINISH) and the client can ask the server to stop sending zlib compressed data with another Telnet control message.

There is a separation option MCCP3 that can be negotiated that works the same way, except it causes data sent from the client to the server to be zlib compressed. I'll note here that this is not widely implemented, as in a typical MUD the sever is probably sending 100x as much data to clients as clients are to the server. If both MCCP2 and MCCP3 are enabled simultaneously each stream has its own zlib dictionary, they aren't shared.

I imagine that in IRC capabilities this would look something like this:

• C: cap ls 302
• S: cap ls :message-tags sasl zlib-server-client zlib-client-server
• C: cap req :message-tags zlib-server-client
• S: cap ack :message-tags zlib-server-client
• From this point on the IRC protocol continues as usual except the underlying TCP stream from the server to the client is zlib compressed

[hhirtz]

Other compression format to consider: https://github.com/facebook/zstd

Why not keep it simple and have one capability to enable compression on both directions?

[SadieCat]

Why is there a need for this exactly? IRC messages are incredibly lightweight compared to other protocols.

We used to support compression for S2S connections (which are much noisier than C2S connections) but it was removed because its essentially a needlesss optimisation.

[RyanSquared]

A lot of clients have issues when connecting to Freenode because they join too many large channels at once. I'd be interested in knowing if the issue is with the amount of data they have to read, or the amount of time it takes to process that data. If it's the first, this may help, but I doubt it personally.

[dequis]

Yeah, those are CPU issues.

Like, in bitlbee we just naively stored user lists in linked lists (because that's the only off-the-shelf data structure we had) and channels with 10k members would make it hang for minutes because linked lists suck at scaling.

In general it seems to be very easy to accidentally implement code that sucks at larger-than-usual channels because those are rare and you don't normally join a thousand channels in your local dev setup.

Just benchmark instead prematurely optimizing parts that are a non-issue.

[jesopo]

A lot of clients have issues when connecting to Freenode because they join too many large channels at once. I'd be interested in knowing if the issue is with the amount of data they have to read, or the amount of time it takes to process that data. If it's the first, this may help, but I doubt it personally.

this is typically them sending too much too quickly (recvq flood) rather than not reading fast enough (sendq flood). it's pretty difficult to trigger a sendq flood

[jwheare]

On paper, this sounds like something that might benefit us at IRCCloud. e.g. if we could switch to compressed streams for all connections to a major IRC network, it might save us a lot of bandwidth usage. However, the CPU cost of decompression on every line might cause more trouble than it's worth.

In situations like a large netsplit where we have to handle massive amounts of quits and parts arriving on all connections in a short space of time, the system can get taxed and backed up processing those lines. The reality is that we don't really pay for bandwidth, it's never something we exhaust, but performance is something that is definitely felt by users.

[dequis]

Obligatory mention of CRIME - which is not trivial to translate from HTTP to IRC (you don't get to trigger hand-crafted requests) but it sounds like a fun puzzle to solve, in many cases it's easy to predict what traffic irc clients send and receive.

[delthas]

Doesn't TLS already have built-in compression? For example there's the standard DEFLATE compression that uses the same compression algorithm as zlib. I haven't made any tests, but it's possibly already enabled and used by popular IRC daemons and clients? I guess you could run openssl s_client -connect host:port or Wireshark your TLS handshake to test it.

If using DEFLATE in TLS is a security issue due to CRIME then compressing with zlib ourselves before passing it to TLS would be an issue as well.

[grawity]

Doesn't TLS already have built-in compression?

Not anymore, no. It was removed as a feature from TLSv1.3, and I faintly remember GnuTLS and LibreSSL removing compression-related code for earlier versions as well.

[delthas]

I just looked into CRIME & BREACH and how they could be applied for IRC. Those exploits work by a) having MITM read access to the victim traffic b) making the client send a secret along with some attacker-crafted data, and checking how compressed the result is. If the secrets and attacker data are similar, then it's likely there was some redundancy. Then by brute-forcing this you can guess the secret in a linear time wrt the secret length (as opposed to exponential time for naive password bruteforcing). That's it.

So for these compression+encryption exploits to be applicable, you need a way to make a client send the secret along with your attacker-crafted data several times (theoretical best estimate: 10 characters * 50 values per char ~= 1000 times). Since you'll get a lot of noise from any other messages sent by the client that will influence the compression algorithm dictionary/sliding window, you'll probably need to run that several times, ignore false positives on guesses like "0000" which are inherently compressed and several other problems, so you'll end up with many many more tries than the theoretical estimate.

Now is there any way that we can make the client send a secret and our crafted guess several thousands times? In this threat model we're not able to impersonate the server, so the best we can do is connect a client to the same IRCd as the victim, and send some specially crafted messages to that client that would it send the secret. AFAIK the only way to make a client send an automatic reply to you as a client are CTCP requests, and the only one where you can choose what the client sends is CTCP PING: send a CTCP PING secret_guess padding to a client and it will reply with PING secret_guess padding. This solution already has an issue: most clients do some kind of CTCP throttling and blacklisting, and servers do some message throttling anyway so you can't go too fast.

What secrets could we possibly steal? In any case it must be something that doesn't change often and will be sent several times, so reading regular private messages is out of the question. To me there are 2 such secrets: account passwords (SASL / NickServ), channel keys.

Account passwords are likely only sent once per connection, or very rarely. Likewise channel keys are only sent once per connection, or when leaving the channel and joining again, etc.

To make a compression-based attack work on those, you would need to make/wait for the client to send that secret, then send it a CTCP PING, then checking the size of the compressed record containing the PING response. So for account passwords, wait for a client to identify to NickServ (likely done right after registering to the IRCd), then send it a PING. For channel keys, wait for the client to join the channel (likely done right after registering) then send a PING. However that's only for one try, and we need several thousands of them! Which means you'd need to make the client connect, reply to your PING, disconnect, then try again, several thousand times. Disconnecting a client is likely possible by sending an invalid TLS record if you're actively MITMing. If you're only passively MITMing without injecting packets this can't work. So the attack would probably look like this:

1. the victim connects to the server
2. the victim sends PRIVMSG NickServ IDENTIFY secret_password post-registration
3. the attacker makes a rogue client connected to the same IRCd send a CTCP PING secret_guess padding to the victim
4. the victim replies to the PING
5. the attacker watches the victim traffic, looks at the size of the compressed+encrypted PING response, checks if it's very small compared to the secret_guess
6. the attacker sends an invalid TLS record from the server to make the victim disconnect
7. the victim disconnects from the server and probably waits a bit before reconnecting
8. goto :1

That's the probably the best you can do. However it has so many issues that make it impractical:

• servers will heavily throttle connections
• clients will heavily throttle reconnect attempts
• clients will heavily throttle CTCP messages
• everything is extremely unreliable and you'll get so many false positives because the compressed message size is influenced by:
• • the last messages sent by the client
• • your secret guess inherent redundancy (eg 0000)
• • your secret guess being similar to other messages the client sent (eg USER will probably be very compressed)
• the attack will take very, very long (rough estimate: 15s reconnect attempt throttling * 10000 tries = 2 days at best!!)
• (any user will notice these disconnections and reconnect attempts)
• (you need to have an active MITM access to your victim)

So to me, compression+encryption based attacks are not applicable to IRC in practice. It is completely safe from a security POV to add compression to all server messages, and safe in practice for all client messages.

[DarthGandalf]

In this threat model we're not able to impersonate the server

Why not? There are still open wifi spots in cafes, where someone could poison DNS or ARP and make client traffic go through MITM. Even for IRC.

[RyanSquared]

Why not? There are still open wifi spots in cafes, where someone could poison DNS or ARP and make client traffic go through MITM. Even for IRC.

But doesn't encryption solve the problem of server impersonation by requiring a valid certificate?

[progval]

AFAIK the only way to make a client send an automatic reply to you as a client are CTCP requests,

You forgot about bots.

And clients with OTR, which auto-replies to you (and you control your own nick, which the victim's client sends).

But doesn't encryption solve the problem of server impersonation by requiring a valid certificate?

What ratio of users have a client that verifies certificates by default?

[RyanSquared]

What ratio of users have a client that verifies certificates by default?

I don't know, but we could list it as a security concern for clients that want to implement compression.

[slingamn]

What ratio of users have a client that verifies certificates by default?

In the threat model we're discussing (where the attacker can inject an invalid TLS record to break the connection), the attacker could just MITM such a client directly and steal the data.

[delthas]

Right. If you're able to modify all packets and the client doesn't verify the TLS certificate, then security is gone, regardless of compression. We should not even list that as a security concern for implementing compression.

[delthas]

You forgot about bots.

And clients with OTR, which auto-replies to you (and you control your own nick, which the victim's client sends).

Right. These two are ways to make the victim possibly send a controlled reply in an automated manner. However it doesn't make the attack any easier: the main issue is making the client send the secret you want to steal, which is likely a password sent only on connection. Even if you find another way to make the client send you a specific message, like those 2, the only way you can make a client send the secret is to force it to reconnect. So yeah, the client might have a nicer throttling on bot messages or OTR messages but you still have to make them disconnect and reconnect to the IRCd several thousand times.

[slingamn]

The main concern to me is account passwords (I don't think channel keys are a very robust security measure). This seems substantially mitigated by using SASL instead of NICKSERV IDENTIFY, and then by recommending that clients negotiate SASL before negotiating compression (which prevents the password or password-derived content from being incorporated into the compression dictionary).

It's also worth noting that the original proposal has separate capabilities for compressing data sent by the server and data sent by the client. If we only compress data sent by the server, that seems to eliminate the CRIME/BEAST problem of leaking secrets controlled by the client. (Of course, the attack also applies in principle to data sent by the server, but we don't have a convincing proof of concept for such an attack on another protocol.)

I think it would be very interesting to get some preliminary performance estimates here for the bandwidth vs. CPU tradeoff (maybe playing with different zstd levels).

[delthas]

Why is there a need for this exactly? IRC messages are incredibly lightweight compared to other protocols.

I think the main advantage here is reducing mobile data usage for clients running on smartphones. I've just run a quick test on my bouncer: it uses approxmately 4~ MB per day for its upstream connections. In the future, with TAGMSG and +typing, with some additional tags like msgid, time, with CHATHISTORY and aggressive caching, with regular wireless mobile intermittent disconnect and reconnections (because of unreliable wireless mobile connections) and the slightly verbose registering and initial bookkeeping (eg WHO #chan123456), it will only amount to more data used, especially in the next generation of modern IRC apps for mobile phones.

Most of my IRC techy friends don't use their smartphones so much and have a low-cost, 50MB/month plan. If IRC uses 5MB/day, or even 2MB/day then that's already too much data for them. It's definitely not a theoretical issue!

It'd be interesting to get some benchmarks for that, but nowadays compressing 5 or even 10 MB per day per user seems trivial CPU-wise. And I think IRC messages will compress really well, even at low-ish zstd levels. (Think all command names, recurring tags, possibly some common channels or nicks...)

[delthas]

Some quick benchmarks, running zstd -b$level log.log (that's zstd benchmark mode), with level between 1 (faster) and 10 (slower) and zstd -b1 --fast=$level log.log for negative levels from -1 to -10, on the last 24 hours raw logs received from one IRCd.

My lscpu: Intel(R) Core(TM) i3-8100 CPU @ 3.60GHz. The benchmark only runs on one thread.

Benchmark format: uncompressed_size -> compressed_size (compression_ratio), encoding_speed, decoding_speed.

Received from freenode:

-19#og_.log          :    444602 ->    225485 (1.972), 765.8 MB/s ,1826.0 MB/s 
-18#og_.log          :    444602 ->    219408 (2.026), 754.5 MB/s ,1762.2 MB/s 
-17#og_.log          :    444602 ->    215734 (2.061), 726.6 MB/s ,1735.1 MB/s 
-16#og_.log          :    444602 ->    211869 (2.098), 715.5 MB/s ,1739.3 MB/s 
-15#og_.log          :    444602 ->    210148 (2.116), 698.5 MB/s ,1712.5 MB/s 
-14#og_.log          :    444602 ->    205290 (2.166), 694.3 MB/s ,1681.6 MB/s 
-13#og_.log          :    444602 ->    198477 (2.240), 676.8 MB/s ,1655.9 MB/s 
-12#og_.log          :    444602 ->    193840 (2.294), 658.9 MB/s ,1632.8 MB/s 
-11#og_.log          :    444602 ->    190806 (2.330), 651.1 MB/s ,1596.4 MB/s 
-10#og_.log          :    444602 ->    182745 (2.433), 639.0 MB/s ,1577.7 MB/s 
-9#log_.log          :    444602 ->    177657 (2.503), 627.2 MB/s ,1566.2 MB/s 
-8#log_.log          :    444602 ->    174050 (2.554), 614.2 MB/s ,1554.0 MB/s 
-7#log_.log          :    444602 ->    168480 (2.639), 604.0 MB/s ,1541.2 MB/s 
-6#log_.log          :    444602 ->    167618 (2.652), 581.3 MB/s ,1512.5 MB/s 
-5#log_.log          :    444602 ->    155850 (2.853), 568.6 MB/s ,1499.0 MB/s 
-4#log_.log          :    444602 ->    153456 (2.897), 546.4 MB/s ,1464.8 MB/s 
-3#log_.log          :    444602 ->    143179 (3.105), 526.1 MB/s ,1436.5 MB/s 
-2#log_.log          :    444602 ->    139622 (3.184), 493.7 MB/s ,1402.4 MB/s 
-1#log_.log          :    444602 ->    121817 (3.650), 451.2 MB/s ,1377.5 MB/s 
 1#log_.log          :    444602 ->     95043 (4.678), 380.9 MB/s ,1115.0 MB/s 
 2#log_.log          :    444602 ->     89433 (4.971), 323.8 MB/s ,1086.3 MB/s 
 3#log_.log          :    444602 ->     85869 (5.178), 260.4 MB/s ,1087.9 MB/s 
 4#log_.log          :    444602 ->     85745 (5.185), 288.2 MB/s ,1082.4 MB/s 
 5#log_.log          :    444602 ->     81445 (5.459), 152.8 MB/s ,1112.0 MB/s 
 6#log_.log          :    444602 ->     80484 (5.524), 136.6 MB/s ,1128.0 MB/s 
 7#log_.log          :    444602 ->     79344 (5.603), 103.8 MB/s ,1166.8 MB/s 
 8#log_.log          :    444602 ->     78920 (5.634),  84.7 MB/s ,1170.6 MB/s 
 9#log_.log          :    444602 ->     78669 (5.652),  67.5 MB/s ,1187.7 MB/s 
10#log_.log          :    444602 ->     78669 (5.652),  68.4 MB/s ,1182.5 MB/s 

Received from Rizon:

-19#og_.log          :    402416 ->    195767 (2.056), 824.3 MB/s ,1883.8 MB/s 
-18#og_.log          :    402416 ->    192606 (2.089), 803.8 MB/s ,1825.9 MB/s 
-17#og_.log          :    402416 ->    190006 (2.118), 788.6 MB/s ,1782.3 MB/s 
-16#og_.log          :    402416 ->    187066 (2.151), 773.1 MB/s ,1775.2 MB/s 
-15#og_.log          :    402416 ->    185499 (2.169), 762.4 MB/s ,1758.4 MB/s 
-14#og_.log          :    402416 ->    184810 (2.177), 764.4 MB/s ,1753.3 MB/s 
-13#og_.log          :    402416 ->    181445 (2.218), 731.5 MB/s ,1703.2 MB/s 
-12#og_.log          :    402416 ->    178154 (2.259), 717.5 MB/s ,1672.9 MB/s 
-11#og_.log          :    402416 ->    175536 (2.292), 699.6 MB/s ,1639.5 MB/s 
-10#og_.log          :    402416 ->    173975 (2.313), 683.5 MB/s ,1603.4 MB/s 
-9#log_.log          :    402416 ->    169770 (2.370), 671.5 MB/s ,1599.8 MB/s 
-8#log_.log          :    402416 ->    166827 (2.412), 652.3 MB/s ,1566.6 MB/s 
-7#log_.log          :    402416 ->    165077 (2.438), 637.2 MB/s ,1563.3 MB/s 
-6#log_.log          :    402416 ->    160577 (2.506), 623.0 MB/s ,1520.5 MB/s 
-5#log_.log          :    402416 ->    155362 (2.590), 590.0 MB/s ,1480.3 MB/s 
-4#log_.log          :    402416 ->    149619 (2.690), 559.1 MB/s ,1437.5 MB/s 
-3#log_.log          :    402416 ->    143252 (2.809), 542.0 MB/s ,1427.0 MB/s 
-2#log_.log          :    402416 ->    138014 (2.916), 505.1 MB/s ,1383.4 MB/s 
-1#log_.log          :    402416 ->    122647 (3.281), 451.4 MB/s ,1362.5 MB/s 
 1#log_.log          :    402416 ->     92984 (4.328), 373.7 MB/s ,1100.5 MB/s
 2#log_.log          :    402416 ->     89318 (4.505), 312.3 MB/s ,1046.2 MB/s 
 3#log_.log          :    402416 ->     85745 (4.693), 241.4 MB/s ,1065.9 MB/s 
 4#log_.log          :    402416 ->     85611 (4.701), 273.9 MB/s ,1064.3 MB/s 
 5#log_.log          :    402416 ->     82373 (4.885), 143.1 MB/s ,1070.3 MB/s 
 6#log_.log          :    402416 ->     81352 (4.947), 131.0 MB/s ,1101.8 MB/s 
 7#log_.log          :    402416 ->     80332 (5.009), 104.4 MB/s ,1138.5 MB/s 
 8#log_.log          :    402416 ->     79678 (5.051),  86.7 MB/s ,1149.5 MB/s 
 9#log_.log          :    402416 ->     79377 (5.070),  69.7 MB/s ,1158.8 MB/s 
10#log_.log          :    402416 ->     79377 (5.070),  69.1 MB/s ,1154.5 MB/s

Received from a server of mine I use with msgid, labeled-response, time, and several clients using TAGMSG +typing. Notice the compression ratio is twice higher (as well as the encoding speed). This is very interesting to me: the extra data usage we get with the latest IRCv3 specs can be heavily compressed by zstd.

-19#og_.log          :    243488 ->     54615 (4.458), 964.3 MB/s ,1805.1 MB/s 
-18#og_.log          :    243488 ->     53050 (4.590), 959.7 MB/s ,1795.4 MB/s 
-17#og_.log          :    243488 ->     53561 (4.546), 946.2 MB/s ,1808.6 MB/s 
-16#og_.log          :    243488 ->     51625 (4.716), 940.8 MB/s ,1789.9 MB/s 
-15#og_.log          :    243488 ->     50379 (4.833), 926.3 MB/s ,1766.7 MB/s 
-14#og_.log          :    243488 ->     49203 (4.949), 930.0 MB/s ,1766.8 MB/s 
-13#og_.log          :    243488 ->     47760 (5.098), 922.0 MB/s ,1758.1 MB/s 
-12#og_.log          :    243488 ->     46122 (5.279), 901.1 MB/s ,1724.1 MB/s 
-11#og_.log          :    243488 ->     44572 (5.463), 903.3 MB/s ,1725.9 MB/s 
-10#og_.log          :    243488 ->     43370 (5.614), 900.2 MB/s ,1723.7 MB/s 
-9#log_.log          :    243488 ->     42813 (5.687), 874.1 MB/s ,1700.8 MB/s 
-8#log_.log          :    243488 ->     41811 (5.824), 874.4 MB/s ,1719.0 MB/s 
-7#log_.log          :    243488 ->     40401 (6.027), 869.4 MB/s ,1716.2 MB/s 
-6#log_.log          :    243488 ->     39091 (6.229), 841.6 MB/s ,1706.2 MB/s 
-5#log_.log          :    243488 ->     38537 (6.318), 824.0 MB/s ,1690.2 MB/s 
-4#log_.log          :    243488 ->     36619 (6.649), 804.2 MB/s ,1694.7 MB/s 
-3#log_.log          :    243488 ->     35044 (6.948), 767.1 MB/s ,1665.9 MB/s 
-2#log_.log          :    243488 ->     33540 (7.260), 736.0 MB/s ,1660.7 MB/s 
-1#log_.log          :    243488 ->     30908 (7.878), 676.0 MB/s ,1633.6 MB/s 
 1#log_.log          :    243488 ->     26980 (9.025), 634.8 MB/s ,1485.6 MB/s 
 2#log_.log          :    243488 ->     25995 (9.367), 542.8 MB/s ,1514.0 MB/s 
 3#log_.log          :    243488 ->     26023 (9.357), 555.4 MB/s ,1486.0 MB/s 
 4#log_.log          :    243488 ->     25616 (9.505), 248.2 MB/s ,1502.1 MB/s 
 5#log_.log          :    243488 ->     24849 (9.799), 231.6 MB/s ,1546.7 MB/s 
 6#log_.log          :    243488 ->     23648 (10.30), 165.1 MB/s ,1673.1 MB/s 
 7#log_.log          :    243488 ->     23224 (10.48), 132.0 MB/s ,1630.5 MB/s 
 8#log_.log          :    243488 ->     22608 (10.77), 107.5 MB/s ,1777.3 MB/s 
 9#log_.log          :    243488 ->     22395 (10.87),  83.5 MB/s ,1789.1 MB/s 
10#log_.log          :    243488 ->     22303 (10.92),  60.9 MB/s ,1805.9 MB/s

Sent data from the same server, harder to compress but there's much fewer data sent by a client than sent by the server so that's OK:

-19#og_.log          :     27171 ->     12147 (2.237),1577.3 MB/s ,3178.6 MB/s 
-18#og_.log          :     27171 ->     11963 (2.271),1423.1 MB/s ,2801.1 MB/s 
-17#og_.log          :     27171 ->     11422 (2.379),1431.8 MB/s ,2920.4 MB/s 
-16#og_.log          :     27171 ->     11123 (2.443),1326.2 MB/s ,2745.9 MB/s 
-15#og_.log          :     27171 ->     11086 (2.451),1307.9 MB/s ,2649.3 MB/s 
-14#og_.log          :     27171 ->     10641 (2.553),1243.3 MB/s ,2512.3 MB/s 
-13#og_.log          :     27171 ->     10352 (2.625),1190.1 MB/s ,2476.6 MB/s 
-12#og_.log          :     27171 ->      9973 (2.724),1161.4 MB/s ,2371.8 MB/s 
-11#og_.log          :     27171 ->      9823 (2.766),1129.0 MB/s ,2380.5 MB/s 
-10#og_.log          :     27171 ->      9471 (2.869),1126.1 MB/s ,2141.5 MB/s 
-9#log_.log          :     27171 ->      9475 (2.868),1084.2 MB/s ,2289.6 MB/s 
-8#log_.log          :     27171 ->      9067 (2.997),1133.8 MB/s ,2317.4 MB/s 
-7#log_.log          :     27171 ->      8878 (3.060),1095.2 MB/s ,2305.4 MB/s 
-6#log_.log          :     27171 ->      8484 (3.203),1034.3 MB/s ,2148.8 MB/s 
-5#log_.log          :     27171 ->      8078 (3.364), 936.0 MB/s ,2029.2 MB/s 
-4#log_.log          :     27171 ->      7958 (3.414), 862.8 MB/s ,2014.2 MB/s 
-3#log_.log          :     27171 ->      7540 (3.604), 787.5 MB/s ,1910.1 MB/s 
-2#log_.log          :     27171 ->      7312 (3.716), 637.6 MB/s ,1830.9 MB/s 
-1#log_.log          :     27171 ->      6691 (4.061), 486.6 MB/s ,1579.2 MB/s 
 1#log_.log          :     27171 ->      5495 (4.945), 432.5 MB/s ,1210.6 MB/s 
 2#log_.log          :     27171 ->      5472 (4.965), 473.6 MB/s ,1140.2 MB/s 
 3#log_.log          :     27171 ->      5393 (5.038), 395.1 MB/s ,1150.9 MB/s 
 4#log_.log          :     27171 ->      5351 (5.078), 389.7 MB/s ,1074.0 MB/s 
 5#log_.log          :     27171 ->      5101 (5.327), 172.1 MB/s ,1065.2 MB/s 
 6#log_.log          :     27171 ->      4977 (5.459), 123.1 MB/s ,1106.0 MB/s 
 7#log_.log          :     27171 ->      4946 (5.494),  99.1 MB/s ,1130.8 MB/s 
 8#log_.log          :     27171 ->      4910 (5.534),  87.1 MB/s ,1154.5 MB/s 
 9#log_.log          :     27171 ->      4889 (5.558),  76.3 MB/s ,1180.1 MB/s 
10#log_.log          :     27171 ->      4870 (5.579),  64.1 MB/s ,1186.3 MB/s

There's a wide range of speed presets to choose from depending on your server load. I think level 1 is the sweet spot if you can afford that CPU usage, and higher levels are not optimal because they slow down encoding & decoding for small returns. What's the average data throughput of IRCd nodes in the wild? If a server sends less than 100MB/s then it needs less than 30% of a single CPU core to compress all streams through zstd level 1. Compressing is really OK performance-wise to me, and a compression ratio of 4 to 8 is worth it.

[SadieCat]

As I said previously regardless of whether compression is safe why is it necessary? There are InspIRCd users with very large instances and I've seen their performance stats and network bandwidth has never been an issue. This is just finding a solution for a problem that doesn't exist.

[delthas]

As I said previously regardless of whether compression is safe why is it necessary?

See my previous comment

[SadieCat]

See my previous comment

I saw it and I don't think that your friends are really representative of the average user. Please provide evidence that this is necessary.

[DarthGandalf]

See my previous comment

IRCv3 moves pretty slowly, so I think by the time your proposal will be implemented, your friends will already have better internet.

[hhirtz]

I saw it and I don't think that your friends are really representative of the average user. Please provide evidence that this is necessary.

It's alright. This idea can start from a vendored spec to provide actual feedback. Both sides showed their arguments on the question of the validity of such idea. If you don't have anything more to say on the matter, let's just skip to how this could be specified?

To recap: compression is advertised with a capability (instead of an ISUPPORT token with a command or something, it would not allow the registration burst to be compressed). However, IIUC it is preferable to not compress the SASL negotiation, from a security POV? In this case, should compression be negotiated, it would be enabled when registration has finished (001 being the first compressed message), instead of when the capability is ACKed.

Second point: how do clients choose the compression algorithm? They can't REQ with a capability argument. I suggest the negociation to be à la SASL:

S: CAP * LS :encoding=gzip,zstd
C: CAP REQ encoding
S: CAP ACK encoding
C: ENCODE gzip
S: ENCODE gzip
... finish registration
S: *gzipped 001*

I chose "encoding" to match the "Accept-Encoding" HTTP header, but you see what I mean. The servers ACKs the compression request by simply echoing the ENCODE command.

Some people want to compress only one side of the connection (I suppose to increase security when using NickServ?). The ENCODE command could accept more parameters.

More points to be solved: should the client be allowed to request compression after it has registered? Should it be allowed to disable it at any moment?

[dequis]

Some quick benchmarks, running zstd -b$level log.log (that's zstd benchmark mode), with level between 1 (faster) and 10 (slower) and zstd -b1 --fast=$level log.log for negative levels from -1 to -10, on the last 24 hours raw logs received from one IRCd.

You'll want to account for a lack of buffering. Compressing a file containing a day's worth of logs is easy, compressing 50-100 bytes that you need to send right now is not.

For a srcSize over 256kb (like your files), level 1 of zstd means a window size of 512kbyte (src). The minimum window size is 1kbyte. Frames have overhead.

For a bunch of messages, it might have less overhead to send uncompressed frames.

[slingamn]

However, IIUC it is preferable to not compress the SASL negotiation, from a security POV? In this case, should compression be negotiated, it would be enabled when registration has finished (001 being the first compressed message), instead of when the capability is ACKed.

I think clients can work around this issue effectively simply by negotiating SASL before compression (and then enabling compression immediately after negotiation). This would allow the registration burst to be compressed.

[slingamn]

Belatedly, I'm also interested in compression as a mitigation for my performance concerns about multiline messages, which were not addressed within the multiline specification itself).

[slingamn]

Second point: how do clients choose the compression algorithm? They can't REQ with a capability argument.

This comes up a lot and it might be time to start thinking about specifying it. Here's what I'd propose:

1. For now, servers publish support for draft/compression=zstd (i.e., we hardcode the current best-of-breed algorithm; I'm assuming it will turn out to be zstd)
2. For now, clients check that the CAP LS value includes zstd and then send CAP REQ draft/compression; no argument means zstd
3. At some point, we specify requesting capability values and allow servers to publish multiple values (e.g., draft/compression=zstd,zmagic). CAP REQ without an argument continues to mean zstd. Clients MAY request any compression algorithm in the list. Servers that publish multiple values MUST respect the client's requested value. Servers can desupport zstd and stop publishing it; this will be detected by legacy clients, since they're already reading the CAP LS value.

[awfulcooking]

Worth mentioning that deflate (gzip) works with websocket clients like Gamja and Kiwi, when the client (most browsers) and server (most reverse proxies) support rfc7692.

Proxies can handle compression at the edge.

TLSv1.3 deprecated compression in general because of CRIME, so yeah whether this is desirable still remains unclear.

However, by default, browsers and e.g. nginx do seem to negotiate websocket compression today.

[slingamn]

I was recently evaluating zstandard vs. zlib in a different context. I found that in a client-server streaming data model, zstandard has a substantial disadvantage relative to zlib at its default settings, in that it consumes much more memory. In the implementation I evaluated, even at the lowest compression level, a 4 MB buffer is recommended:

https://github.com/klauspost/compress/blob/94ad1f0bf03c0761f50c69f9242f49fbb9ab08c0/zstd/encoder_options.go#L221-L232

This would be prohibitive for most server implementations. Decreasing the buffer size below the recommended value is possible, but may impact both CPU performance and compression ratio. So it's not clear how much of an edge zstandard has in this use case. This post goes into some more detail on the design tradeoffs:

https://fastcompression.blogspot.com/2016/04/working-with-streaming.html

[delthas]

Some numbers :smile:

I have let my bouncer run for a day or two. Here's the compression gains for multiple zstd compression levels and window size.

This uses one zstd context per connection, and flushes the buffer after each message is written (the simplest possible implementation; we could batch... batches before flushing).

By buffer I mean the total size needed for compression, per connection.

S->C 02448/06044 kB @ 40.50% means server to client, compressed size is 2448kB, uncompressed size is 6044kB, and therefore the compression ratio is 40.50%

As expected the size of server to client data is >10x the size of client to server data.

The time column can be ignored.

Level 1 - Window 10 - Buffer 0095kB - S->C 02448/06044 kB @ 40.50% - C->S 00106/00235 kB @ 45.31% - Time: 0.4238
Level 1 - Window 11 - Buffer 0102kB - S->C 02320/06044 kB @ 38.40% - C->S 00106/00235 kB @ 45.11% - Time: 0.3424
Level 1 - Window 12 - Buffer 0115kB - S->C 02238/06044 kB @ 37.04% - C->S 00106/00235 kB @ 45.11% - Time: 0.3488
Level 1 - Window 13 - Buffer 0142kB - S->C 02161/06044 kB @ 35.76% - C->S 00106/00235 kB @ 45.11% - Time: 0.3843
Level 1 - Window 14 - Buffer 0196kB - S->C 02106/06044 kB @ 34.85% - C->S 00106/00235 kB @ 45.11% - Time: 0.3468
Level 1 - Window 15 - Buffer 0304kB - S->C 02061/06044 kB @ 34.11% - C->S 00106/00235 kB @ 45.11% - Time: 0.3141
Level 1 - Window 16 - Buffer 0521kB - S->C 02017/06044 kB @ 33.37% - C->S 00106/00235 kB @ 45.11% - Time: 0.3096
Level 1 - Window 17 - Buffer 0953kB - S->C 01985/06044 kB @ 32.84% - C->S 00106/00235 kB @ 45.11% - Time: 0.3211
Level 1 - Window 18 - Buffer 1081kB - S->C 01978/06044 kB @ 32.74% - C->S 00106/00235 kB @ 45.11% - Time: 0.3784
Level 1 - Window 19 - Buffer 1337kB - S->C 01978/06044 kB @ 32.74% - C->S 00106/00235 kB @ 45.11% - Time: 0.3355
Level 1 - Window 20 - Buffer 1849kB - S->C 01978/06044 kB @ 32.74% - C->S 00106/00235 kB @ 45.11% - Time: 0.3236
Level 2 - Window 10 - Buffer 0287kB - S->C 02403/06044 kB @ 39.76% - C->S 00106/00235 kB @ 45.32% - Time: 0.3326
Level 2 - Window 11 - Buffer 0294kB - S->C 02279/06044 kB @ 37.71% - C->S 00106/00235 kB @ 45.13% - Time: 0.3641
Level 2 - Window 12 - Buffer 0307kB - S->C 02202/06044 kB @ 36.43% - C->S 00106/00235 kB @ 45.12% - Time: 0.3394
Level 2 - Window 13 - Buffer 0334kB - S->C 02129/06044 kB @ 35.23% - C->S 00106/00235 kB @ 45.12% - Time: 0.3343
Level 2 - Window 14 - Buffer 0388kB - S->C 02076/06044 kB @ 34.35% - C->S 00106/00235 kB @ 45.12% - Time: 0.3568
Level 2 - Window 15 - Buffer 0496kB - S->C 02031/06044 kB @ 33.60% - C->S 00106/00235 kB @ 45.12% - Time: 0.3524
Level 2 - Window 16 - Buffer 0713kB - S->C 01983/06044 kB @ 32.81% - C->S 00106/00235 kB @ 45.12% - Time: 0.3607
Level 2 - Window 17 - Buffer 1145kB - S->C 01946/06044 kB @ 32.21% - C->S 00106/00235 kB @ 45.12% - Time: 0.3650
Level 2 - Window 18 - Buffer 1273kB - S->C 01936/06044 kB @ 32.03% - C->S 00106/00235 kB @ 45.12% - Time: 0.3304
Level 2 - Window 19 - Buffer 1529kB - S->C 01936/06044 kB @ 32.03% - C->S 00106/00235 kB @ 45.12% - Time: 0.3586
Level 2 - Window 20 - Buffer 2041kB - S->C 01936/06044 kB @ 32.03% - C->S 00106/00235 kB @ 45.12% - Time: 0.3975
Level 3 - Window 10 - Buffer 0799kB - S->C 02366/06044 kB @ 39.15% - C->S 00105/00235 kB @ 44.97% - Time: 0.4038
Level 3 - Window 11 - Buffer 0806kB - S->C 02242/06044 kB @ 37.11% - C->S 00105/00235 kB @ 44.81% - Time: 0.4255
Level 3 - Window 12 - Buffer 0819kB - S->C 02165/06044 kB @ 35.82% - C->S 00105/00235 kB @ 44.81% - Time: 0.4047
Level 3 - Window 13 - Buffer 0846kB - S->C 02090/06044 kB @ 34.58% - C->S 00105/00235 kB @ 44.81% - Time: 0.3802
Level 3 - Window 14 - Buffer 0900kB - S->C 02035/06044 kB @ 33.68% - C->S 00105/00235 kB @ 44.81% - Time: 0.3856
Level 3 - Window 15 - Buffer 1008kB - S->C 01989/06044 kB @ 32.92% - C->S 00105/00235 kB @ 44.81% - Time: 0.3988
Level 3 - Window 16 - Buffer 1225kB - S->C 01939/06044 kB @ 32.09% - C->S 00105/00235 kB @ 44.81% - Time: 0.3844
Level 3 - Window 17 - Buffer 1657kB - S->C 01899/06044 kB @ 31.43% - C->S 00105/00235 kB @ 44.81% - Time: 0.3826
Level 3 - Window 18 - Buffer 1785kB - S->C 01888/06044 kB @ 31.24% - C->S 00105/00235 kB @ 44.81% - Time: 0.3863
Level 3 - Window 19 - Buffer 2041kB - S->C 01888/06044 kB @ 31.24% - C->S 00105/00235 kB @ 44.81% - Time: 0.3828
Level 3 - Window 20 - Buffer 2553kB - S->C 01888/06044 kB @ 31.24% - C->S 00105/00235 kB @ 44.81% - Time: 0.3872
Level 4 - Window 10 - Buffer 2079kB - S->C 02366/06044 kB @ 39.15% - C->S 00105/00235 kB @ 44.97% - Time: 0.5045
Level 4 - Window 11 - Buffer 2086kB - S->C 02242/06044 kB @ 37.11% - C->S 00105/00235 kB @ 44.81% - Time: 0.5208
Level 4 - Window 12 - Buffer 2099kB - S->C 02165/06044 kB @ 35.82% - C->S 00105/00235 kB @ 44.81% - Time: 0.4848
Level 4 - Window 13 - Buffer 2126kB - S->C 02090/06044 kB @ 34.58% - C->S 00105/00235 kB @ 44.81% - Time: 0.5343
Level 4 - Window 14 - Buffer 2180kB - S->C 02035/06044 kB @ 33.68% - C->S 00105/00235 kB @ 44.81% - Time: 0.4942
Level 4 - Window 15 - Buffer 2288kB - S->C 01989/06044 kB @ 32.92% - C->S 00105/00235 kB @ 44.81% - Time: 0.4365
Level 4 - Window 16 - Buffer 2505kB - S->C 01939/06044 kB @ 32.08% - C->S 00105/00235 kB @ 44.81% - Time: 0.4235
Level 4 - Window 17 - Buffer 2937kB - S->C 01899/06044 kB @ 31.42% - C->S 00105/00235 kB @ 44.81% - Time: 0.4766
Level 4 - Window 18 - Buffer 3065kB - S->C 01887/06044 kB @ 31.23% - C->S 00105/00235 kB @ 44.81% - Time: 0.4653
Level 4 - Window 19 - Buffer 3321kB - S->C 01887/06044 kB @ 31.22% - C->S 00105/00235 kB @ 44.81% - Time: 0.4751
Level 4 - Window 20 - Buffer 3833kB - S->C 01887/06044 kB @ 31.22% - C->S 00105/00235 kB @ 44.81% - Time: 0.4400
Level 5 - Window 10 - Buffer 3103kB - S->C 02289/06044 kB @ 37.87% - C->S 00105/00235 kB @ 44.77% - Time: 0.6324
Level 5 - Window 11 - Buffer 3110kB - S->C 02190/06044 kB @ 36.23% - C->S 00104/00235 kB @ 44.64% - Time: 0.6413
Level 5 - Window 12 - Buffer 3123kB - S->C 02113/06044 kB @ 34.97% - C->S 00104/00235 kB @ 44.62% - Time: 0.6887
Level 5 - Window 13 - Buffer 3150kB - S->C 02046/06044 kB @ 33.85% - C->S 00104/00235 kB @ 44.62% - Time: 0.6478
Level 5 - Window 14 - Buffer 3204kB - S->C 01994/06044 kB @ 32.99% - C->S 00104/00235 kB @ 44.61% - Time: 0.6407
Level 5 - Window 15 - Buffer 3312kB - S->C 02050/06044 kB @ 33.92% - C->S 00107/00235 kB @ 45.62% - Time: 0.6021
Level 5 - Window 16 - Buffer 3529kB - S->C 01998/06044 kB @ 33.07% - C->S 00107/00235 kB @ 45.62% - Time: 0.5707
Level 5 - Window 17 - Buffer 3961kB - S->C 01957/06044 kB @ 32.38% - C->S 00107/00235 kB @ 45.62% - Time: 0.6378
Level 5 - Window 18 - Buffer 4089kB - S->C 01947/06044 kB @ 32.22% - C->S 00107/00235 kB @ 45.62% - Time: 0.5430
Level 5 - Window 19 - Buffer 4345kB - S->C 01947/06044 kB @ 32.22% - C->S 00107/00235 kB @ 45.62% - Time: 0.5566
Level 5 - Window 20 - Buffer 4857kB - S->C 01947/06044 kB @ 32.22% - C->S 00107/00235 kB @ 45.62% - Time: 0.5422
Level 6 - Window 10 - Buffer 3103kB - S->C 02284/06044 kB @ 37.80% - C->S 00105/00235 kB @ 44.76% - Time: 0.7222
Level 6 - Window 11 - Buffer 3110kB - S->C 02184/06044 kB @ 36.14% - C->S 00104/00235 kB @ 44.62% - Time: 0.6911
Level 6 - Window 12 - Buffer 3123kB - S->C 02106/06044 kB @ 34.85% - C->S 00104/00235 kB @ 44.61% - Time: 0.6787
Level 6 - Window 13 - Buffer 3150kB - S->C 02037/06044 kB @ 33.70% - C->S 00104/00235 kB @ 44.60% - Time: 0.6749
Level 6 - Window 14 - Buffer 3204kB - S->C 01983/06044 kB @ 32.82% - C->S 00104/00235 kB @ 44.60% - Time: 0.6973
Level 6 - Window 15 - Buffer 3312kB - S->C 02039/06044 kB @ 33.73% - C->S 00107/00235 kB @ 45.61% - Time: 0.5936
Level 6 - Window 16 - Buffer 3529kB - S->C 01987/06044 kB @ 32.88% - C->S 00107/00235 kB @ 45.61% - Time: 0.5893
Level 6 - Window 17 - Buffer 3961kB - S->C 01945/06044 kB @ 32.19% - C->S 00107/00235 kB @ 45.61% - Time: 0.5578
Level 6 - Window 18 - Buffer 4089kB - S->C 01936/06044 kB @ 32.03% - C->S 00107/00235 kB @ 45.61% - Time: 0.5545
Level 6 - Window 19 - Buffer 4345kB - S->C 01936/06044 kB @ 32.03% - C->S 00107/00235 kB @ 45.61% - Time: 0.5526
Level 6 - Window 20 - Buffer 4857kB - S->C 01936/06044 kB @ 32.03% - C->S 00107/00235 kB @ 45.61% - Time: 0.5442

As can be seen, at level 1 you stop getting better compression ratio after Window log 13, which can be understood because message similarities are often found between one line and the next, which are often <2048 bytes apart. That's a memory impact of ~150kB per connection, for a compression ratio of approximately 1/3.

[slingamn]

Nice!

between one line and the next, which are often <2048 bytes apart

That seems high; is this due to client-only tags?

Would it be possible to get comparable numbers for zlib (on its highest setting)?

[delthas]

Would it be possible to get comparable numbers for zlib (on its highest setting)?

Yes! See below for zstd and zlib numbers. No dictionaries used.

zstd - Level 1 - Window 10 - Buffer 0095kB - S->C 02449/06044 kB @ 40.52% - C->S 00107/00235 kB @ 45.88% - Time: 0.4824
zstd - Level 1 - Window 11 - Buffer 0102kB - S->C 02322/06044 kB @ 38.42% - C->S 00107/00235 kB @ 45.68% - Time: 0.3722
zstd - Level 1 - Window 12 - Buffer 0115kB - S->C 02240/06044 kB @ 37.06% - C->S 00107/00235 kB @ 45.67% - Time: 0.3380
zstd - Level 1 - Window 13 - Buffer 0142kB - S->C 02162/06044 kB @ 35.78% - C->S 00107/00235 kB @ 45.67% - Time: 0.3481
zstd - Level 1 - Window 14 - Buffer 0196kB - S->C 02108/06044 kB @ 34.88% - C->S 00107/00235 kB @ 45.67% - Time: 0.3779
zstd - Level 1 - Window 15 - Buffer 0304kB - S->C 02062/06044 kB @ 34.13% - C->S 00107/00235 kB @ 45.67% - Time: 0.3966
zstd - Level 1 - Window 16 - Buffer 0521kB - S->C 02018/06044 kB @ 33.39% - C->S 00107/00235 kB @ 45.67% - Time: 0.3425
zstd - Level 1 - Window 17 - Buffer 0953kB - S->C 01986/06044 kB @ 32.87% - C->S 00107/00235 kB @ 45.67% - Time: 0.3535
zstd - Level 1 - Window 18 - Buffer 1081kB - S->C 01980/06044 kB @ 32.76% - C->S 00107/00235 kB @ 45.67% - Time: 0.3357
zstd - Level 1 - Window 19 - Buffer 1337kB - S->C 01980/06044 kB @ 32.76% - C->S 00107/00235 kB @ 45.67% - Time: 0.3530
zstd - Level 1 - Window 20 - Buffer 1849kB - S->C 01980/06044 kB @ 32.76% - C->S 00107/00235 kB @ 45.67% - Time: 0.3555
zstd - Level 2 - Window 10 - Buffer 0287kB - S->C 02404/06044 kB @ 39.78% - C->S 00107/00235 kB @ 45.89% - Time: 0.4139
zstd - Level 2 - Window 11 - Buffer 0294kB - S->C 02280/06044 kB @ 37.73% - C->S 00107/00235 kB @ 45.70% - Time: 0.4300
zstd - Level 2 - Window 12 - Buffer 0307kB - S->C 02203/06044 kB @ 36.45% - C->S 00107/00235 kB @ 45.69% - Time: 0.3740
zstd - Level 2 - Window 13 - Buffer 0334kB - S->C 02130/06044 kB @ 35.25% - C->S 00107/00235 kB @ 45.69% - Time: 0.3917
zstd - Level 2 - Window 14 - Buffer 0388kB - S->C 02077/06044 kB @ 34.37% - C->S 00107/00235 kB @ 45.69% - Time: 0.4178
zstd - Level 2 - Window 15 - Buffer 0496kB - S->C 02032/06044 kB @ 33.63% - C->S 00107/00235 kB @ 45.69% - Time: 0.3549
zstd - Level 2 - Window 16 - Buffer 0713kB - S->C 01984/06044 kB @ 32.83% - C->S 00107/00235 kB @ 45.69% - Time: 0.3793
zstd - Level 2 - Window 17 - Buffer 1145kB - S->C 01948/06044 kB @ 32.23% - C->S 00107/00235 kB @ 45.69% - Time: 0.3934
zstd - Level 2 - Window 18 - Buffer 1273kB - S->C 01937/06044 kB @ 32.06% - C->S 00107/00235 kB @ 45.69% - Time: 0.3940
zstd - Level 2 - Window 19 - Buffer 1529kB - S->C 01937/06044 kB @ 32.06% - C->S 00107/00235 kB @ 45.69% - Time: 0.3657
zstd - Level 2 - Window 20 - Buffer 2041kB - S->C 01937/06044 kB @ 32.06% - C->S 00107/00235 kB @ 45.69% - Time: 0.3618
zstd - Level 3 - Window 10 - Buffer 0799kB - S->C 02368/06044 kB @ 39.18% - C->S 00107/00235 kB @ 45.54% - Time: 0.4805
zstd - Level 3 - Window 11 - Buffer 0806kB - S->C 02244/06044 kB @ 37.13% - C->S 00106/00235 kB @ 45.38% - Time: 0.4749
zstd - Level 3 - Window 12 - Buffer 0819kB - S->C 02166/06044 kB @ 35.85% - C->S 00106/00235 kB @ 45.37% - Time: 0.4773
zstd - Level 3 - Window 13 - Buffer 0846kB - S->C 02091/06044 kB @ 34.60% - C->S 00106/00235 kB @ 45.37% - Time: 0.4447
zstd - Level 3 - Window 14 - Buffer 0900kB - S->C 02037/06044 kB @ 33.70% - C->S 00106/00235 kB @ 45.37% - Time: 0.4333
zstd - Level 3 - Window 15 - Buffer 1008kB - S->C 01991/06044 kB @ 32.94% - C->S 00106/00235 kB @ 45.37% - Time: 0.4530
zstd - Level 3 - Window 16 - Buffer 1225kB - S->C 01941/06044 kB @ 32.11% - C->S 00106/00235 kB @ 45.37% - Time: 0.4385
zstd - Level 3 - Window 17 - Buffer 1657kB - S->C 01901/06044 kB @ 31.45% - C->S 00106/00235 kB @ 45.37% - Time: 0.4398
zstd - Level 3 - Window 18 - Buffer 1785kB - S->C 01889/06044 kB @ 31.26% - C->S 00106/00235 kB @ 45.37% - Time: 0.4077
zstd - Level 3 - Window 19 - Buffer 2041kB - S->C 01889/06044 kB @ 31.26% - C->S 00106/00235 kB @ 45.37% - Time: 0.4008
zstd - Level 3 - Window 20 - Buffer 2553kB - S->C 01889/06044 kB @ 31.26% - C->S 00106/00235 kB @ 45.37% - Time: 0.4212
zstd - Level 4 - Window 10 - Buffer 2079kB - S->C 02368/06044 kB @ 39.18% - C->S 00107/00235 kB @ 45.54% - Time: 0.5518
zstd - Level 4 - Window 11 - Buffer 2086kB - S->C 02244/06044 kB @ 37.13% - C->S 00106/00235 kB @ 45.38% - Time: 0.5095
zstd - Level 4 - Window 12 - Buffer 2099kB - S->C 02166/06044 kB @ 35.85% - C->S 00106/00235 kB @ 45.37% - Time: 0.5158
zstd - Level 4 - Window 13 - Buffer 2126kB - S->C 02091/06044 kB @ 34.60% - C->S 00106/00235 kB @ 45.37% - Time: 0.5244
zstd - Level 4 - Window 14 - Buffer 2180kB - S->C 02036/06044 kB @ 33.70% - C->S 00106/00235 kB @ 45.37% - Time: 0.5769
zstd - Level 4 - Window 15 - Buffer 2288kB - S->C 01990/06044 kB @ 32.94% - C->S 00106/00235 kB @ 45.37% - Time: 0.6577
zstd - Level 4 - Window 16 - Buffer 2505kB - S->C 01940/06044 kB @ 32.10% - C->S 00106/00235 kB @ 45.37% - Time: 0.6647
zstd - Level 4 - Window 17 - Buffer 2937kB - S->C 01900/06044 kB @ 31.44% - C->S 00106/00235 kB @ 45.37% - Time: 0.5166
zstd - Level 4 - Window 18 - Buffer 3065kB - S->C 01888/06044 kB @ 31.25% - C->S 00106/00235 kB @ 45.37% - Time: 0.5516
zstd - Level 4 - Window 19 - Buffer 3321kB - S->C 01888/06044 kB @ 31.25% - C->S 00106/00235 kB @ 45.37% - Time: 0.5489
zstd - Level 4 - Window 20 - Buffer 3833kB - S->C 01888/06044 kB @ 31.25% - C->S 00106/00235 kB @ 45.37% - Time: 0.4915
zstd - Level 5 - Window 10 - Buffer 3103kB - S->C 02290/06044 kB @ 37.89% - C->S 00106/00235 kB @ 45.34% - Time: 0.7320
zstd - Level 5 - Window 11 - Buffer 3110kB - S->C 02191/06044 kB @ 36.25% - C->S 00106/00235 kB @ 45.21% - Time: 0.6903
zstd - Level 5 - Window 12 - Buffer 3123kB - S->C 02115/06044 kB @ 34.99% - C->S 00106/00235 kB @ 45.19% - Time: 0.7152
zstd - Level 5 - Window 13 - Buffer 3150kB - S->C 02047/06044 kB @ 33.88% - C->S 00106/00235 kB @ 45.18% - Time: 0.6611
zstd - Level 5 - Window 14 - Buffer 3204kB - S->C 01995/06044 kB @ 33.01% - C->S 00106/00235 kB @ 45.18% - Time: 0.7009
zstd - Level 5 - Window 15 - Buffer 3312kB - S->C 02051/06044 kB @ 33.95% - C->S 00108/00235 kB @ 46.19% - Time: 0.6667
zstd - Level 5 - Window 16 - Buffer 3529kB - S->C 02000/06044 kB @ 33.09% - C->S 00108/00235 kB @ 46.19% - Time: 0.6890
zstd - Level 5 - Window 17 - Buffer 3961kB - S->C 01958/06044 kB @ 32.40% - C->S 00108/00235 kB @ 46.19% - Time: 0.6206
zstd - Level 5 - Window 18 - Buffer 4089kB - S->C 01948/06044 kB @ 32.24% - C->S 00108/00235 kB @ 46.19% - Time: 0.6261
zstd - Level 5 - Window 19 - Buffer 4345kB - S->C 01948/06044 kB @ 32.24% - C->S 00108/00235 kB @ 46.19% - Time: 0.6361
zstd - Level 5 - Window 20 - Buffer 4857kB - S->C 01948/06044 kB @ 32.24% - C->S 00108/00235 kB @ 46.19% - Time: 0.6275
zstd - Level 6 - Window 10 - Buffer 3103kB - S->C 02286/06044 kB @ 37.82% - C->S 00106/00235 kB @ 45.33% - Time: 0.8264
zstd - Level 6 - Window 11 - Buffer 3110kB - S->C 02186/06044 kB @ 36.16% - C->S 00106/00235 kB @ 45.19% - Time: 0.8240
zstd - Level 6 - Window 12 - Buffer 3123kB - S->C 02108/06044 kB @ 34.87% - C->S 00106/00235 kB @ 45.17% - Time: 0.8112
zstd - Level 6 - Window 13 - Buffer 3150kB - S->C 02038/06044 kB @ 33.72% - C->S 00106/00235 kB @ 45.16% - Time: 0.7764
zstd - Level 6 - Window 14 - Buffer 3204kB - S->C 01985/06044 kB @ 32.84% - C->S 00106/00235 kB @ 45.16% - Time: 0.7458
zstd - Level 6 - Window 15 - Buffer 3312kB - S->C 02040/06044 kB @ 33.75% - C->S 00108/00235 kB @ 46.17% - Time: 0.6490
zstd - Level 6 - Window 16 - Buffer 3529kB - S->C 01988/06044 kB @ 32.90% - C->S 00108/00235 kB @ 46.17% - Time: 0.6243
zstd - Level 6 - Window 17 - Buffer 3961kB - S->C 01946/06044 kB @ 32.21% - C->S 00108/00235 kB @ 46.17% - Time: 0.6452
zstd - Level 6 - Window 18 - Buffer 4089kB - S->C 01937/06044 kB @ 32.05% - C->S 00108/00235 kB @ 46.17% - Time: 0.6391
zstd - Level 6 - Window 19 - Buffer 4345kB - S->C 01937/06044 kB @ 32.05% - C->S 00108/00235 kB @ 46.17% - Time: 0.5823
zstd - Level 6 - Window 20 - Buffer 4857kB - S->C 01937/06044 kB @ 32.05% - C->S 00108/00235 kB @ 46.17% - Time: 0.6192
zlib - Level 1 -           - Buffer 0138kB - S->C 05124/06044 kB @ 84.78% - C->S 00290/00235 kB @ 123.46% - Time: 1.8593
zlib - Level 2 -           - Buffer 0139kB - S->C 01809/06044 kB @ 29.94% - C->S 00095/00235 kB @ 40.70% - Time: 1.2551
zlib - Level 3 -           - Buffer 0141kB - S->C 01792/06044 kB @ 29.65% - C->S 00095/00235 kB @ 40.52% - Time: 1.3004
zlib - Level 4 -           - Buffer 0145kB - S->C 01757/06044 kB @ 29.08% - C->S 00095/00235 kB @ 40.64% - Time: 1.3591
zlib - Level 5 -           - Buffer 0153kB - S->C 01729/06044 kB @ 28.61% - C->S 00095/00235 kB @ 40.48% - Time: 1.4058
zlib - Level 6 -           - Buffer 0169kB - S->C 01713/06044 kB @ 28.35% - C->S 00093/00235 kB @ 39.96% - Time: 1.4238
zlib - Level 7 -           - Buffer 0202kB - S->C 01708/06044 kB @ 28.27% - C->S 00093/00235 kB @ 39.93% - Time: 1.8129
zlib - Level 8 -           - Buffer 0268kB - S->C 01704/06044 kB @ 28.19% - C->S 00093/00235 kB @ 39.93% - Time: 1.7097
zlib - Level 9 -           - Buffer 0399kB - S->C 01703/06044 kB @ 28.19% - C->S 00093/00235 kB @ 39.93% - Time: 1.7473

Extract with lines of similar buffer sizes:

zstd - Level 1 - Window 13 - Buffer 0142kB - S->C 02162/06044 kB @ 35.78% - C->S 00107/00235 kB @ 45.67% - Time: 0.3481
zlib - Level 3 -           - Buffer 0141kB - S->C 01792/06044 kB @ 29.65% - C->S 00095/00235 kB @ 40.52% - Time: 1.3004

From a quick glance at the data, zlib compresses slightly better at identical buffer sizes but does look slower. (I used zstd from C->Go bindings whereas zlib is a pure Go impl so maybe that affects the performance difference)

[nektro]

briefly mentioned above, i agree compression might see its biggest headways in CHATHISTORY results or other/future operations that return bulk data.