Open jcus0006 opened 10 months ago
Some updates:
Tried a run with no workers on the client/scheduler node. Each worker takes around 2gb of RAM or maybe even a bit more. By doing this I ensured that the contact tracing is at least attempted. While it didn't crash with an out of memory "killed" error it still crashed around the same area of the first clean-up attempt. This is because the logs for the 3rd day of the itinerary and the contact network are present but the ones for the contract tracing / contacts clean-up are not. There were 2 contacts to be contact traced, and for this reason, one worker was going to to be started. But the contacts that had to be sent over were 750 mb for the directcontacts, and 450mb each for the indexes, which amount to 1.65gb.
The contact tracing stage seems to have run properly; seems to have run something for every worker, which is unnecessary but did not take a lot of time; and took 337 seconds. However, something happened around this time such that the Dask client became unusable and it ruined the rest of the process. Some extra logs need to be put in place to figure out what caused this.