Open bwesterb opened 1 year ago
requires message of many GB in memory. Not sure if this is gonna be realistic since usually large reads are done in batches of smaller sizes.
Those tests don't use a buffer of GBs. Instead they use the size suggested by MaxWriteSize()
, which atm is lanes CPUs 8kB, ie. 192kB on M2 Pro.
but timings make me think if something must be done differently (or I missed something)
There clearly is some concurrency bottleneck here which I haven't figured out yet.
Use options style API, so that the common case is very simple:
but we can provide options elegantly:
Allows multithreaded computation with the WithWorkers() option. On M2 Pro scales well with a few workers, but isn't able to utilize all cores effectively:
We only reach 23GB/s (at 12x) instead of the lower bound of 33GB/s expected with 10 performance cores.
Adds {Max,Next}WriteSize to suggest the caller how big to choose their Write() calls.