mattjala
commented
4 months ago Testing results from a benchmark on my local machine, capped to 8 threads to avoid overwhelming HSDS with requests:
Elapsed multi time to read from 12,000,000 elems in 64 datasets = 15.2272
Elapsed serial time to read from 12,000,000 elems in 64 datasets = 12.7379
Elapsed multi time to write to 12,000,000 elems in 64 datasets = 12.1155
Elapsed serial time to write to 12,000,000 elems in 64 datasets = 15.0920CPU usage in HSDS was only capped out for the SN during the multi write. Other than that, this benchmark isn't CPU-capped (at least running locally). That or the GIL might be why the performance gain is only ~25% for writing, and is actually a performance penalty for reading
Here are some other benchmarks with different numbers of parallel threads:
4 Threads
Elapsed multi time to read from 12000000 elems in 64 datasets = 13.1364
Elapsed serial time to read from 12000000 elems in 64 datasets = 14.3920
Elapsed multi time to write to 12000000 elems in 64 datasets = 19.3822
Elapsed serial time to write to 12000000 elems in 64 datasets = 18.4147
2 Threads
Elapsed multi time to read from 12000000 elems in 64 datasets = 11.3716
Elapsed serial time to read from 12000000 elems in 64 datasets = 14.4107
Elapsed multi time to write to 12000000 elems in 64 datasets = 13.8106
Elapsed serial time to write to 12000000 elems in 64 datasets = 17.8859
1 Thread (Sanity Check)
Elapsed multi time to read from 12000000 elems in 64 datasets = 14.9388
Elapsed serial time to read from 12000000 elems in 64 datasets = 14.8551
Elapsed multi time to write to 12000000 elems in 64 datasets = 19.8820
Elapsed serial time to write to 12000000 elems in 64 datasets = 20.4815
High-level MultiManager interface allows users to use multiple threads to perform reads/writes on a collection of (distinct) datasets. Examples of how to use this interface are shown in the
TestMultiManagerclass.Resolves #166