nodejs / help

:sparkles: Need help with Node.js? File an Issue here. :rocket:
1.47k stars 280 forks source link

worker_threads is used by many people,but we can only use SharedArrayBuffer(for shared memory) ,but Buffer is node's first citizen(socket eg),can you share Buffer to SharedArrayBuffer ? #4288

Closed introspection3 closed 3 months ago

introspection3 commented 11 months ago

What is the problem this feature will solve?

threads_worker is used by many people,but we can only use SharedArrayBuffer ,but Buffer is node's first citizen(socket eg),can you share Buffer to SharedArrayBuffer ?

What is the feature you are proposing to solve the problem?

let instance=SharedArrayBuffer.from(Buffer instance);//zero copy let instance2=Buffer.From(SharedArrayBuffer instance);//zero copy

What alternatives have you considered?

No response

aduh95 commented 11 months ago

You mean node:worker_threads, right? If all buffers could be shared, the language would not have bothered to come up with a SharedArrayBuffer implementation. What you can do is the following:

function sabToBuffer(sab) {
  return new Uint8Array(sab);
}

Uint8Array is not exactly a Buffer, but I don't think you'll find any Node.js API that accept Buffer and not Uint8Array (if you do, that'd be a bug, please report it).

On the other hand, if you have a Buffer/Uint8Array and want to transfer its content to an other thread, you have two options:

// Option 1: copy the content to a SharedArrayBuffer:
function copyBufferToSAB(buffer, sab) {
  if (buffer.byteLength > sab.byteLength) throw new Error('Buffer overflow');
  new Uint8Array(sab).set(buffer);
}

// Option 2: transfer the underlying array buffer:
function transferBuffer(buffer) {
  otherThread.postMessage(buffer, [buffer.buffer]);
}

Option 2 is way faster, but note that only one thread interact with a non-sharable buffer, meaning that once it is transfered, the original thread can no longer have access to that buffer data.

introspection3 commented 11 months ago

@aduh95 no,you are wrong. copy ArrayBuffer is so expensive。

introspection3 commented 11 months ago

@aduh95 do you know zero copy

aduh95 commented 11 months ago

no,you are wrong. copy ArrayBuffer is so expensive

I don't know why you say I'm wrong, I don't think I have said anything about copy not being expensive. Like I said, transfering is the faster option, that's the one that let you reuse the same data on another thread without copying data.

RedYetiDev commented 3 months ago

@introspection3 @aduh95 is this issue resolved? If so, feel free to close the issue!

Thanks!