CMCDragonkai
commented
2 years ago Preliminary load testing has already been done for EFS. Of course we can see that workers are quite a bit slow, and for single core usage, it's never worth it to use workers. It's only when there are many other concurrent operations where it becomes worth it. The exact calculation is located in another issue I forgot about.
For now I'm going to close this issue until we have a more pressing need to benchmark.
Right now we only have benchmarks for crypto operations and the DB operations.
To properly assess what is the optimal block size to use (probably more than 4 KiB, but definitely less than 1 MiB), we have to do a proper load test on the EFS instance.
This requires performing a sequential set of operations vs parallel set of operations.
With the tests done in https://github.com/MatrixAI/js-polykey/issues/209, we can see that due to the speed for encryption/decryption, using a worker thread will always be slower cause calling workers have 1.5ms overhead.
The speed up can only happen when the there are multiple worker threads being used, so when there are multiple parallel operations occurring either through batching and/or multiple EFS reads/writes.
Benny is not the right tool to do load testing, but we may be able to adapt it for proper load testing.
Otherwise we can use a better load testing tool like artillery, but it needs to be easy to integrate, and not go through the network. This is not an HTTP server. It's a library.
Remember as long as the main thread is doing other things, and that the work is more expensive than call overhead divided by the number of workers, then we should have a profitable use of worker threads.
Additional Context
Tasks
WorkerManagerinsideEncryptedFS