search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

'ragavi-vis' -- Unexpected Interruption During Plotting -- 'inspect' worker #1558

Open bazhaoyu opened 6 months ago

bazhaoyu commented 6 months ago

Hi,

I'm currently processing MeerKAT data using CARACAL and encountered an unexpected issue with 'ragavi-vis' during the plotting phase. The process gets interrupted, and I am unsure why. Here's the error message I encountered (screenshot attached below): error

My environment is set up with CARACAL version 1.0.7 and STIMELA 1.7.7, using APPTAINER 1.2.4. For further information, here are my configure file and output log file here.

Previously, I successfully processed data of around 5 GB, but this issue arises with a larger dataset of approximately 2 TB. Could this issue be related to the large size of the data? Has anyone experienced something similar, or could you suggest potential solutions? Thanks in advance for your help!

KshitijT commented 6 months ago

@bazhaoyu most probably if this works with smaller datasets the size - probably RAM? might be the issue. @Mulan-94 any suggestions here? Does using shadems instead of ragavi give you similar issues ?

bazhaoyu commented 6 months ago

@bazhaoyu most probably if this works with smaller datasets the size - probably RAM? might be the issue. @Mulan-94 any suggestions here? Does using shadems instead of ragavi give you similar issues ?

Thank you for your reply. I noticed that maybe I could set a larger 'mem_limit' with a value of ~ 80 GB (default is 8GB) for 'ragavi-vis'. I will let you know if it works. If not, I would like to try 'shadems' to see what happens.

bazhaoyu commented 5 months ago

Hi, Using 'ragavi-vis' with an 80 GB memory limit reported the same error. Using 'shadems' reported another error: "NoneType" object has no attribute 'upper'. I am not clear if there are any necessary settings that I didn't set when choosing 'shadems'？

This is the configuration info:

##########
inspect:  
  enable: true
  label_in: cal 
  uvrange: '>150'
  dirname: 'crosscal'
  label_plot: '1gc'
  mem_limit: '80GB'
  shadems:
    enable: true 
    default_column: CORRECTED_DATA 
  standard_plotter: shadems  
  field: calibrators   # target  # bpcal  # gcal 
  real_imag:
    enable: true
    avgtime: '60'
    avgchan: '10'
  amp_chan:
    enable: true
    avgtime: '60'
    avgchan: '1'
  phase_chan:
    enable: true
    avgtime: '60'
    avgchan: '1'
  amp_uvwave:
    enable: true
    avgtime: '60'
    avgchan: '10'
  phase_uvwave:
    enable: true
    avgtime: '60'
    avgchan: '10'
################

This is the screenshot of the error: error1

I also tried the configuration as your sample -- 'meerkat-continuum-defaults.yml', but set the 'standard_plotter: shadems'. However, I got the same error "NoneType" object has no attribute 'upper'.

Do you happen to know what the problem is or what I should do to fix it? Thanks a lot!

Athanaseus commented 2 weeks ago

My environment is set up with CARACAL version 1.0.7 and STIMELA 1.7.7, using APPTAINER 1.2.4. For further information, here are my configure file and output log file here.

Please provide the shadems full log. The link above doesn't work.

Fil8 commented 2 weeks ago

Maybe the ragavi memory issue is due to the fact that averaging in time or in chans is turned off in the code?

https://github.com/caracal-pipeline/caracal/blob/2d338e2afc5ed8fc723be95e74b071890d4a7bed/caracal/workers/inspect_worker.py#L519C1-L520C35