search

ratt-ru / QuartiCal

CubiCal, but with greater power.
MIT License
7 stars 4 forks source link

pandas.errors.InvalidIndexError #336

Closed bngcebetsha closed 5 days ago

bngcebetsha commented 1 week ago

Describe the bug I have a calibrated dataset flagged around 63% from MeerKAT, the output is also on the markdown file attached. I need to correct for the phase on the target field as well as the phase slope.

quartical_error_proxima_cen_target.md

To Reproduce I have attached both commands I ran to derive and apply the gains on the markdown file above

Expected behavior I expect for the derived gain solutions to be applied to the relevant data columns on the measurement set. Such that when I image the corrected data I get improved images.

Version quartical==0.2.0

JSKenyon commented 1 week ago

Hmm, that looks very suspicious. I would definitely recommend updating to the latest version of QuartiCal.

My instinct is that your gain solutions may not have been written correctly. Are you running QuartiCal locally or somewhere I have access to (Rhodes/CHPC boxes)? If so, I can take a quick look.

bngcebetsha commented 1 week ago

Hmm, that looks very suspicious. I would definitely recommend updating to the latest version of QuartiCal.

My instinct is that your gain solutions may not have been written correctly. Are you running QuartiCal locally or somewhere I have access to (Rhodes/CHPC boxes)? If so, I can take a quick look.

I might agree the calibration step ended with a killed - I am running this on young.ru... on this path /home/bngcebetsha/cal_quartical

JSKenyon commented 1 week ago

Kill your run please - your config is completely swamping memory.

JSKenyon commented 1 week ago

The reason this happens is because you haven't set input_ms.time_chunks or input_ms.freq_chunks appropriately, and by default QC is trying to use all the cores on the node (which is something I should change). Setting dask.threads will let you control that. The error you originally reported is simply caused by the fact that the gains were never written (because the job died due to an OOM condition I would guess).

Edit: QC may have been fine but other work on the machine may have led to am OOM death.

bngcebetsha commented 1 week ago

Kill your run please - your config is completely swamping memory.

we have DynSpecMS to thank for this. Ok thanks @JSKenyon . I will try out your suggestions and update/close the issue when done.

JSKenyon commented 1 week ago

Kill your run please - your config is completely swamping memory.

we have DynSpecMS to thank for this. Ok thanks @JSKenyon . I will try out your suggestions and update/close the issue when done.

Great! Sorry - I was struggling to communicate with young so I was just guessing that it was QC to blame. But yeah, swap was full and that usually means game over. Hitting swap is bad enough.