chrisiacovella
commented
11 months ago Multithreading breakdown performance
I ran a few benchmarks with different chunk sizes and threads on my 8 core and 32 core machines, mostly because I was curious how much these impacted the results I showed above. In terms of the number of records to fetch per thread, the sweet spot seems to be 5-10 records. In terms of the number of concurrent threads, 8-10 threads per core.
Fetching 1000 records on 8 core machine
chunk_size n_threads time (s)
1 48 31.9
2 48 26.2
5 48 13.3
10 48 13.7
25 48 16.2
50 48 27.6
chunk_size n_threads time (s)
10 24 25.2
10 48 13.7
10 64 10.9
10 96 11.9
Fetch 2000 records on 8 core machine
chunk_size n_threads time (s)
10 24 43.7
10 48 23.6
10 64 18.4
10 96 16.9
10 128 21.2
Fetch 2000 records on 32 core machine
chunk_size n_threads time (s)
10 24 35.4
10 48 21.1
10 64 20.9
10 96 19.8
10 128 14.5
10 196 11
10 200 9.81
fetch 3000 records on 32 core machine
chunk_size n_threads time (s)
5 24 57.5
5 48 31.2
5 64 27
5 96 26.1
5 128 21.6
5 196 17.7
5 256 15.5
5 288 15.2
5 320 15.4
5 384 11.2
5 416 10.9
5 512 11.5
fetch 3000 records on 32 core machine
chunk_size n_threads time (s)
10 24 52.4
10 48 30.6
10 64 25.7
10 96 24
10 128 19.2
10 196 14.4
10 256 14.1
10 288 15.1
10 300 15
``
bennybp
peastman
This issue relates to downloading large batches of records, such as fetching an entire dataset.
There are two main hurdles I've run into with the batch download: 1- Efficiency of the process 2- Loss of connection to the database during retrieval
Note, this issue relates to issues #740 and #741.
Here, I'm will outline some benchmarks and my attempted solutions.
Serial retrieval of records benchmarks
Currently if I want to fetch records I would do something similar to:
The performance for fetching a small number of records is good:
Since this is a serial process, as might be expected, timing scales with the number of records
Based on this scaling, fetching the whole QM9 dataset (with ~133K records) would take about 13 hours (in practice this is about right).
Multithreaded retrieval bencharmks
I used
concurrent.futuresto wrap theget_recordsfunction above, to allow for concurrent multithreading. I'll note I choseconcurrent.futuresbecause it works in jupyter notebooks without needing to do any sort of "tricks". The code below is a pretty bare bones implementation of multithreaded retrieval. I'll note, this code will chunk together 25 records per thread (i.e.,chunk_size) and allow for 48 concurrent threads (i.e.,n_threads); these values seems to be pretty reasonable on my 8 core MacBook.Benchmarks for fetching records:
Extrapolating to the whole qm9 dataset, it would take about <15 minutes to fetch the 133K records, which seems quite reasonable as compared to 13 hours.
Loss of database connectivity
If I were to up the number of records to, say 10000, at some point during the process I will get the following error:
Note, if I just turn off my wifi (to simulate an internet hiccup), I get the following error:
I modifying the code above (shown below), to simply put the record retrieval in a try/except structure (inside a while loop that will retry a max of 1000 times). I'll note, in the code below, instead of recording the actual data fetched, i just keep track of the number of "failures" to connect to the server.
I had this code fetch 10000 records 3 times; the timing was very consistent at about 100 s.
Interestingly, the number of failures to connect didn't change much between the 3 runs: 405, 407, and 408. If it were just my internet being a bit flaky, I would have expected those to be less consistent. Since data is being broken up into 25 record chunks, that means 400 total threads, so the ~400 failures seems a little suspect . Digging a little into these numbers, each thread on the first call to the portal,ends up in the except statement. This confuses me because, if every single thread needs to reconnect initially, how does the first implementation of the threading, without the try/except of this would even work? (There must be something odd about the try/exception statement that is eluding me). Regardless, this means that 400 of those "failures" aren't real, so we are really dealing with 5, 7, and 8, i.e., <1 failure to connect for every 1000 record calls.
Issue #741 suggests using retry behind the scenes which seems to be a more user friendly approach and would allow for a bit smarter approach in terms of connection retries.
Creating multithreaded
get_records()and 'get_entries()' functions in the qcportal would be quite beneficial.