search

ratt-ru / CubiCal

A fast radio interferometric calibration suite.
GNU General Public License v2.0
18 stars 13 forks source link

Cubical hanging: CRITICAL "'DD' D0T1F5 Significant gain variance in directions 2" #442

Closed Kincaidr closed 3 years ago

Kincaidr commented 3 years ago

Hi,

My cubical is hanging again, i have searched for any Runtime error in log file that resulted in the hanging from last time, could not find any. I am suspecting its one of these CRITICAL errors about "'DD' D0T1F5 Significant gain variance in directions 2" but i am not sure. image

Full log file is attached below

ddcal_0.log

o-smirnov commented 3 years ago

No your trouble lies here:

INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Detected a total of 503.81GiB of system memory.
INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Per-solver (worker) memory use estimated at 53.71GiB: 10.66% of total system memory.
INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Peak I/O memory use estimated at 347.35GiB: 68.94% of total system memory.
INFO      11:59:43 - wisdom             [0.6/0.6 3.5/3.5 0.1Gb] Total peak memory usage estimated at 1690.13GiB: 335.47% of total system memory.

The cure is the usual, fewer workers, smaller chunks.

I also highly recommend enabling --dist-safe, so that it bails out up front on the memory check.