search

apertif / apercal

GNU General Public License v3.0
3 stars 9 forks source link

mfclean never ends #192

Closed tammojan closed 5 years ago

tammojan commented 5 years ago

It seems like we should give mfclean a maximum number of clark iterations. The job below needed to be manually killed after four hours.

After reaching maximum clark iterations we could abort selfcal, as apparently something went wrong. Continuum imaging went ahead fine with the last selfcal result.

03/20/2019 04:35:06 AM - basher - DEBUG : Returning output.
03/20/2019 04:35:06 AM - masher - DEBUG : mfclean cutoff=9.144674550043419e-05 map=01/map_00 region="mask(01/mask_02)" beam=01/beam_00 niters=1000000 model=01/model_01 out=01/model_02 
03/20/2019 04:35:06 AM - basher - DEBUG : mfclean cutoff=9.144674550043419e-05 map=01/map_00 region="mask(01/mask_02)" beam=01/beam_00 niters=1000000 model=01/model_01 out=01/model_02 
03/20/2019 09:26:46 AM - basher - DEBUG : Command = mfclean cutoff=9.144674550043419e-05 map=01/map_00 region="mask(01/mask_02)" beam=01/beam_00 niters=1000000 model=01/model_01 out=01/model_02 
03/20/2019 09:26:46 AM - basher - DEBUG : 

mfclean: Revision 1.16, 2019/01/29 23:03:22 UTC

Sub-beam size is    13 by   159
FFTing the beams ...
Calculating the map*beam ...
Calculating patches ...
Loading the model and getting residuals ...
Starting to iterate ...
 Clark  Iterations: 1000
 Residual min,max,rms:   -5.687E-04   7.453E-04   4.064E-04
 Clark  Iterations: 2000
 Residual min,max,rms:   -5.269E-04   6.768E-04   3.800E-04
 Clark  Iterations: 3000
 Residual min,max,rms:   -5.495E-04   6.250E-04   3.586E-04
 Clark  Iterations: 4000
 Residual min,max,rms:   -5.380E-04   5.851E-04   3.402E-04

======= Many more Clark iteration omitted here ====== 

 Clark  Iterations: 758000
 Residual min,max,rms:   -4.358E+20   4.884E+20    Infinity
 Clark  Iterations: 759000
 Residual min,max,rms:   -4.365E+20   4.893E+20    Infinity

03/20/2019 09:26:46 AM - basher - DEBUG : ### Warning:  Sub-beam size is dangerously small
Terminated

03/20/2019 09:26:46 AM - apercal.pipeline.start_pipeline - WARNING : Failed beam 31, skipping that from scal
03/20/2019 09:26:46 AM - apercal.pipeline.start_pipeline - ERROR : Error in command mfclean: 
### Warning:  Sub-beam size is dangerously small
Terminated
Traceback (most recent call last):
  File "/home/apercal/apercal/apercal/pipeline/start_pipeline.py", line 265, in start_apercal_pipeline
    p4.go()
  File "/home/apercal/apercal/apercal/modules/scal.py", line 117, in go
    self.phase()
  File "/home/apercal/apercal/apercal/modules/scal.py", line 571, in phase
    mfclean.go()
  File "/home/apercal/apercal/apercal/libs/lib.py", line 387, in go
    output = masher(**self.__dict__)
  File "/home/apercal/apercal/apercal/libs/lib.py", line 202, in masher
    out = basher(cmd, showasinfo=False)
  File "/home/apercal/apercal/apercal/libs/lib.py", line 276, in basher
    raise RuntimeError("Error in command " + cmd.split(" ")[0] + ": \n" + err)
RuntimeError: Error in command mfclean: 
### Warning:  Sub-beam size is dangerously small
Terminated

03/20/2019 09:26:46 AM - apercal.modules.continuum - INFO : Starting CONTINUUM IMAGING
tammojan commented 5 years ago

This was in beam 31 of 190311152; on happili-04:/data/apertif/190311152/apercal.log

tammojan commented 5 years ago

Ah the error is in the command. niters=1000000 is maybe not the best value. Changing it to 50000.