Closed streamich closed 1 year ago
Here is a simple benchmark, which shows how slow creating a buffer slice using a buffer-like objects is:
Note:
new Uint8Array
or Uint8Array.prototype.subarray
are the fastest methods I know to create a "slice". But they are still 20x slower than new Slice()
in Node v20.Uint8Array
is getting slower.ArrayBuffer
is in s
, not ms
Buffer
is not in the benchmark, but is about the same to a bit slower than Uint8Array
.Please add to your benchmark the new Node v20 ArrayBuffer resize method https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer/resize
Thanks @e3dio, good to know about ArrayBuffer.ptototype.resize()
, unfortunately it is the worst performing one:
And it does not solve the problem, as:
offset
is always 0
, which is not the case.ArrayBuffer
, which also is not the case, the message encoder will share one large ArrayBuffer
, encode multiple messages into it and hand out the slices.ArrayBuffer.ptototype.resize()
would solve this problem, it is available only in the next Node/V8 version; which will not get wide adoption until a couple of years.Also, I just realized that this benchmark is not a good one for the .resize()
method; as it resizes it "up" as well :)
Created a better benchmark for .resize()
. In this benchmark the buffer is only resized down:
Still, .resize()
is about as slow as creating a new Uint8Array
. About an order of magnitude slower than creating a Slice
.
But more importantly, even if .resize()
was fast, it fails on these points:
- That method assumes that offset is always 0, which is not the case.
- That method assumes that you own the whole ArrayBuffer, which also is not the case, the message encoder will share one large ArrayBuffer, encode multiple messages into it and hand out the slices.
res.endFast(buf, offset, length)
would be even faster than the new Slice()
method. Essentially if we only added these two methods, it would solve 90% of the problem:
interface HttpResponse {
endFast(buf: ArrayBuffer, offset: number, length: number): void;
// Always 3 arguments, does not close connection.
}
interface WebSocket {
sendFast(buf: ArrayBuffer, offset: number, length: number): void;
// Always 3 arguments, binary message, default compression.
}
You want to use buffer as a shared buffer, but how do you deal with race conditions when the uWS reads your buffer are you already overwriting this buffer slice in the node process?
For example
Yes creating a resizable ArrayBuffer is slow and resize is also slow, that is disappointing. This is the fastest way I see, I don't see this in your benchmark:
const buf = Buffer.allocUnsafe(1000); // initial fast buffer
const buf2 = buf.subarray(0,500); // zero copy buffer view of data
Update: new Uint8Array() is slightly faster for Buffer see https://github.com/uNetworking/uWebSockets.js/issues/894#issuecomment-1529103753
The subarray
method returns a ref to a buffer slice, this is not a copy of the buffer, as I wrote above, without solving the race problems, you will not be able to use one buffer for many responses, I advise you to study the performance of the new method that returns a copy of the buffer
uWS.js send/end/write takes a Buffer, there is nothing mentioned about a Buffer view not working, I think uWS copies data on method call so buf.subarray() should work which is very fast
@e3dio .subarray()
is very slow. It is almost identical to new Uint8Array()
and you can see performance of that in the screenshots above.
@uasan yes, my understanding is that uWS makes a copy of the passed in data. .copyBytesFrom
is not going to help here, the whole idea is to copy bytes and allocate new buffer-like objects as little as possible.
.subarray() is very slow. It is almost identical to new Uint8Array()
I don't see subarray in your benchmark
@e3dio included .subarray()
:
Creating a temporary Uint8Array
just to represent a "slice" is a lot of unnecessary memory usage, see in this StackOverflow answer how much fields each Uint8Array
instance holds.
In summary, each string consumes
5*8 = 40
bytes, each typed array consumes26*8 = 208
bytes.
Essentially, creating a temporary Uint8Array
using .subarray()
or new Uint8Array()
is extra 208 bytes that need to be allocated and then immediately garbage collected.
In, something like
response.endFast(uint8, offset, length);
socket.sendFast(uint8, offset, length);
no need to allocate extra 208 bytes. (A WebSocket message itself could be way less than this 208 bytes overhead.)
For 100K messages per second, that is extra 20.8 MB/s of work for V8 allocator and garbage collector.
Just to summarize my case:
encoder
, which encodes responses into binary format.ArrayBuffer
of the message.
Uint8Array
.ArrayBuffer
just for your message, it will be oversized. Because encoders initially allocate more space as they don't know the size of encoded data ahead of time..subarray()
which is very slow and allocates 208 bytes for every message.uws.get('/rpc', (res, req) => {
const result = rpc.exec(/* ... */);
const slice = encoder.encode(result);
// very slow:
res.end(slice.buf.subarray(slice.start, slice.end));
// would be nice to have:
res.endFast(slice.buf, slice.start, slice.end);
})
@e3dio I'm not sure I understand what you mean. Here, Buffer.prototype.subarray()
is even 2x slower than Uint8Array.prototype.subarray()
:
Also, Buffer
is legacy. Everyone is moving away from it to use Uint8Array
instead. There is this myth that Buffer.allocUnsafe()
is fast because it has a shared pool of 4KB sized Buffer
s. But I haven't been able to see any performance benefit of tapping into that pool.
Added res.end()
case to show the performance impact of res.end(new Slice())
over res.end(buf, offset, length)
:
If you need a node Buffer
there is a little trick you an use:
const FastBuffer = Buffer[Symbol.species]
const buf2 = new FastBuffer(buf1.buffer, buf1.byteOffset + offset, length) // buf1.subarray(offset ,length)
Or something along those lines.
as compared to what the Node.js core developers have done
To be fair... this is a problem with UInt8Array
which Node doesn't implement, it's V8.
Using the cork
handler is important for performance, which means you need to take care of slice immutability before calling cork, i.e. you will have to create temporary states like semophores, which in the end will load the GC more than 208 bytes )
151 ms to create 10 million slices does not sound like a problem. Say that you will be capped at 300k req/sec at best - that would put the total overhead of slicing at 0.4% if I did my math correctly. This is not a bottleneck, esp. not considering how sluggish JavaScript is in comparison.
The math is correct, assuming 300K req/sec, V8 will alloc/dealloc ~60 MB/sec unnecessarily and spend 0.45% or 4.6ms each second being blocked on creating slices.
That is assuming the developer will use the most efficient .subarray()
method, if they use something less efficient, say some .subarray()
on some other prototype or, .resize()
or .slice()
on ArrayBuffer
or something else, it could easily be 2x (or more) slower. Which would be 0.9%, almost 1%.
Would you accept a PR if I implemented something like below?
interface WebSocket {
sendSpecialOrSomethingLikeThat(buffer: ArrayBuffer, offset: number, length: number): void;
}
@streamich here is good benchmark I think
Results:
Uint8Array-ab: 8s (cheating: did not start from Buffer or Uint8Array) Uint8Array-arr: 20s Uint8Array-buf: 20s subarray-arr: 15s subarray-buf: 23s
So if you start from Buffer, new Uint8Array()
is 15% faster than subarray()
at 20s vs 23s. If you start from Uint8Array, subarray()
is 50% faster than new Uint8Array()
at 15s vs 20s. My encoding library uses Buffer so I would use new Uint8Array()
If we're talking an overhead of 0.4% and ~200 bytes per slice I don't really see how this is even remotely an issue. It just sounds like a natural property of a scripting language. Scripting languages aren't zero overhead.
The existing uWS.js API should be fine. Use subarray or new Uint8Array as described above to send uWS.js a zero copy slice/view of buffer
There is a distinction between what is in scope and what is not. We can't add bypasses for custom use cases such as slice-and-send or extract-and-send or receive-to-json or anything like that. Then we would just add a bunch of hacks to workaround shortcomings of the language. Either accept the shortcomings of scripting or swap to the C++ or C library if you need guaranteed lowest overhead.
0.4% is not even a low hanging fruit
In my RPC requests and responses have schema, and encoder is JIT compiled according to that schema, which writes straight to big shared Uint8Array
, so there are almost no allocations. Overall the encoding is almost 10x faster than Buffer.from(JSON.stringify(data))
, effectively this makes this last 0.5% bit of .subarray()
the slowest part in the whole encoding process, which I wanted to eliminate.
I think you have effectively optimized your process to the maximum, and you are safe to move on to something else ;) Also shameless plug check out my encoding library for smallest possible size, speed, and ease of use https://github.com/e3dio/packBytes Although I need to update the Buffer sizing mechanism, currently it calculates the exact buffer size needed instead of estimating and sending a view as described in this issue, I need to update that to improve speed
@e3dio thanks, yeah will take a look.
In my RPC requests and responses have schema, and encoder is JIT compiled according to that schema, which writes straight to big shared
Uint8Array
, so there are almost no allocations. Overall the encoding is almost 10x faster thanBuffer.from(JSON.stringify(data))
, effectively this makes this last 0.5% bit of.subarray()
the slowest part in the whole encoding process, which I wanted to eliminate.
I totally salute your dedication but JavaScript is not the right place to be if you care about 0.4% slicing overhead.
There is also the overhead of GC of all the temporaries which might not be represented in the benchmarks...
Typically generational GC do not promote young objects that die young (that's the GC way of doing stack variables) to full on GC'd objects
uWebSockets.js accepts
ArrayBuffer
-and-co JavaScript binary formats when writing to the socket, that is great, but is not enough.All data encoders pre-allocate more memory than needed, but current API does not allow to effectively provide only the slice of the final encoded data, instead of creating a new buffer-like object.
A simplified example:
Instead, responses could be sped up substantially (and consume less memory), with an API as follows:
, where
0
is offset andsize
is the length of binary chunk to copy. (No call to.subarray()
, which is very slow.)Constructing
ArrayBuffer
, orUint8Array
, orBuffer
are very, very slow in Node.js and every release, at least since Node v14, it is getting slower.I would like to propose an optimized API, where the binary chunk slice can be specified using
offset
andlength
params, instead of creating a temporaryArrayBuffer
orUint8Array
.Below are three options for this new API.
Option 1
This would be the fastest option:
The
.end()
and.write()
are the most performance critical methods, so.writeHeader()
and.writeStatus()
could be ignored here.Option 2
This option is slower, but would integrate nicely with the existing API:
constructing a new instance of
Slice
is about 30x faster than creating a "slice" usingArrayBuffer
orUint8Array
.In TypeScript:
This option would integrate well also with the
.writeHeader
and.writeStatus
methods.uWebSockets
could provide theSlice
class, which V8 would hopefully compile as hidden class:Option 3
This option is similar to Option 2, but instead of extending the
RecognizedString
type, various existing methods would receive optionaloffset
andlength
parameters:The same applies to the WebSocket instances.