I've noticed this on my tqdm-branch but not sure if it is actually an issue or just honestly expected based of how simulating each UVTask is not the same. However I have been experimenting with an MWA phase II simulation with the following parameters
Nbls: 9
Ntimes: 117
Nfreqs: 256
Nsrcs: 744672
Tasks: 26687232.0
I noticed with 30 PUs that initially ~800 UVTasks are processed per second, but this quickly slows to 400-500 and eventually settles near 200 for a long time. Near the end of the simulation it only reports ~20 UVTasks are being processed per second. Before finishing. I have noticed something odd where the simulation ended when the progress bar was only at 94% but that may just be that the estimated Ntasks_tot was a large over estimate for the simulation :thinking:
I've noticed this on my tqdm-branch but not sure if it is actually an issue or just honestly expected based of how simulating each UVTask is not the same. However I have been experimenting with an MWA phase II simulation with the following parameters
Ntasks_totwas a large over estimate for the simulation :thinking: