I have an application that needs to support high pubsub throughput where the id's are cryptographic hashes so naturally binary channels and patterns are a great fit.
However it doesn't look like ioredis supports binary patterns with psubscribe or pmessageBuffer, and has some issues with binary channels in general.
2. Internally Buffer channels and patterns are serialized with utf8 which is not binary safe
It seems for equality (eg storing in Sets) that all binary (NodeJS Buffer) channels and patterns are converted to utf8 string instead of a binary safe string like latin1 (ISO-8859-1).
For example of how this can cause issues with binary data, here are two byte sequences whose utf8 conversions from NodeJS Buffer collide, despite the Buffer instances containing different byte sequences.
to use binary safe strings (eg bufferInstance.toString('latin1') instead of bufferInstance.toString() or bufferInstance.toString('utf8') where serializing buffers to strings is necessary for equality (eg Maps and Sets)
to support passing a Buffer instance for the pattern argument into psubscribe
to return back a Buffer instance instead of string for the pattern argument of listener in redis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer)
Hello :wave:
I have an application that needs to support high pubsub throughput where the id's are cryptographic hashes so naturally binary channels and patterns are a great fit.
However it doesn't look like
ioredis
supports binary patterns withpsubscribe
orpmessageBuffer
, and has some issues with binary channels in general.Problems
1.
redis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer)
providesutf8
pattern instead ofBuffer
instanceReceived
pmessage
messages have their pattern converted toutf8
with no option to return the original Buffer instead.See https://github.com/redis/ioredis/blob/ec42c82ceab1957db00c5175dfe37348f1856a93/lib/DataHandler.ts#L143C1
2. Internally
Buffer
channels and patterns are serialized withutf8
which is not binary safeIt seems for equality (eg storing in Sets) that all binary (NodeJS
Buffer
) channels and patterns are converted toutf8
string instead of a binary safe string likelatin1
(ISO-8859-1).See https://github.com/redis/ioredis/blob/ec42c82ceab1957db00c5175dfe37348f1856a93/lib/DataHandler.ts#L160
For example of how this can cause issues with binary data, here are two byte sequences whose
utf8
conversions from NodeJSBuffer
collide, despite theBuffer
instances containing different byte sequences.What am I looking for?
bufferInstance.toString('latin1')
instead ofbufferInstance.toString()
orbufferInstance.toString('utf8')
where serializing buffers to strings is necessary for equality (eg Maps and Sets)Buffer
instance for thepattern
argument intopsubscribe
Buffer
instance instead of string for thepattern
argument oflistener
inredis.on('pmessageBuffer', listener(pattern: string, channel: Buffer, payload: Buffer)