Meeting 06/11/2023

[jcus0006]

• Fix memory issue that doesn't allow the simulation to run until the end (delete unnecessary data or persist into CSV - see if pandas as data storage is more efficient than Numpy for this case)
• Find out why load balancing is not effective (keep in mind that the strategy I am using may not be optimal)
• Order the assignment of work to nodes by slowest first (as part of load balancing)
• Checkpointing (be able to continue the simulation from a particular day - this is useful when trying out different scenarios)
• Send the simulation to Prof Abela to try out on the super computer (previous run was tested on the mega computer, this is even faster) could make the argument that with a single super computer we will always be faster. but if having multiple small computers, there is still added value from the distributed version (must make sure that the latest changes work with multiprocessing)
• Trying it out at the University lab will make the dissertation more complete (Prof Abela will speak to the University lab administrator to check about the virtualization technology being used and whether Hyper-V can be used. He will also check whether he can get funding to buy 6 SSDs which we can clone with the same image, install in the lab computers and then format when ready)
• Check release notes of later Python versions and see if it is worth updating

[jcus0006]

Some updates:

1. Made various attempts to fix memory usage. Found various issues along the way. Started making use of a "start day" marker to reduce memory usage in the directcontacts array by removing the rather redundant "day", and keep a range as a dict, e.g. {1: 0, 2: 3560} indicating that day 1 starts from 0 and day 2 starts from 3560. Then implemented a clean up of the array, but it means that both the start date index, as well as the agent1 and agent2 indexes need to be re-computed after removing the first "n" elements. Also fixed some bugs that I found in the contact tracing base method along the way. After all this, it was taking a rather long amount of time. Initially I thought it was the clean-up + re-computation of the indexes, however, it only takes around 20-30 seconds, which in the grand scheme of things, is probably acceptable. The time is being lost in the binary search, and not because it is slow, but because of how large the directcontact array becomes. For e.g. up until day 2, the array had 4.3 million contacts, and was 500mb in size.
2. This was being tested on purpose, taking into consideration 48 hours of contact tracing in mind, which essentially requires the current day and 2 previous days to be persisted. The clean-up requires some intermediate / working memory and it seems to be using up all the memory available on the client/scheduler node while cleaning up. Another attempt will be made using 24 hours of contact tracing, which should theoretically delay the memory from getting out of hand.
3. It was also noted that the itinerary is taking longer to start than the contact network, and this is probably because of the agents_dynamic structure that is being split and sent to all the workers. An exercise should be made to confirm that all the data that is being sent and returned in this regard is actually necessary.
4. Fixed an issue with the contact tracing whereby the agents to be contact traced were being added into the contact_tracing_agent_ids set when starting quarantine from the itinerary, but also when scheduling a test result that ends up in the agent being quarantined from the contact network (epidemiological part). The latter part was extra, and has since been removed.
5. By this fix, the contact tracing is likely to start around the 8-10 day mark, which is not feasible for testing purposes with the 500k population. The 1k population test is not feasible to test the contact tracing. Using a 10k population could be a feasible compromise. Otherwise, a workaround needs to be planned out, e.g. checkpointing.
6. Contact tracing has been re-implemented to use Dask distribution. Figuring out the necessity of this requires more testing.
7. Contact tracing with distribution is likely to be very slow, because of the size of the directcontacts array (and the indexes). In this case it might be required to first send the array once; and then each worker loads it from a file into memory. The same approach can be considered for the other methods too, maybe it applies.
8. It might be required that the directcontacts and the indexes arrays are persisted as files at the end of every simulation day to save up memory space that grows like crazy.
9. I have also changed the implementation of the CellType and SimCellType slightly. These were being stored in memory as strings, e.g. "household" and "residence", for cell type and sim cell type, respectively. Now using Enums, they should utilise less memory space.
10. The other points in the original description are yet to be considered.

[jcus0006]

Tried a run with no workers on the client/scheduler node. Each worker takes around 2gb of RAM or maybe even a bit more. By doing this I ensured that the contact tracing is at least attempted. While it didn't crash with an out of memory "killed" error it still crashed around the same area of the first clean-up attempt. This is because the logs for the 3rd day of the itinerary and the contact network are present but the ones for the contract tracing / contacts clean-up are not. There were 2 contacts to be contact traced, and for this reason, one worker was going to to be started. But the contacts that had to be sent over were 750 mb for the directcontacts, and 450mb each for the indexes, which amount to 1.65gb.

The contact tracing stage seems to have run properly; seems to have run something for every worker, which is unnecessary but did not take a lot of time; and took 337 seconds. However, something happened around this time such that the Dask client became unusable and it ruined the rest of the process. Some extra logs need to be put in place to figure out what caused this.