search

lofar-astron / factor

Facet calibration for LOFAR
http://www.astron.nl/citt/facet-doc
GNU General Public License v2.0
19 stars 12 forks source link

"re_image" steps fails #154

Closed rvweeren closed 7 years ago

rvweeren commented 8 years ago

The re-imaging for one of my facet fails (this was the facet where I peeled 3C295). Seems to be related to the masking, but it is entirely clear what the problem precisely is...

2016-11-06 16:38:48 DEBUG facetimage_facet_patch_539.executable_args: Results for job 0 submitted by ('127.0.0.1', 36054) 2016-11-06 16:38:48 INFO node.lofar.executable_args: Total time 2215.3182s; user time: 19862.2722s; system time: 2428.2441s 2016-11-06 16:38:48 DEBUG node.lofar.executable_args: Start time was 1478466113.2628s; end time was 1478468328.5815s 2016-11-06 16:38:48 DEBUG facetimage_facet_patch_539.executable_args: 2016-11-06 16:38:48 WARNING facetimage_facet_patch_539.executable_args: 2016-11-06 16:38:48 INFO facetimage_facet_patch_539.executable_args: Subprocess completed with exit status 0: /bin/sh -c python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/executable_args.py 0 127.0.1.1 54170 2016-11-06 16:38:48 DEBUG facetimage_facet_patch_539.executable_args: compute.dispatch results job 0: job_duration: 2215.53727889, returncode: 0 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Adding node_logging_information 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Writing data map file: /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/mapfiles/wsclean_image_full1.mapfile 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Writing data map file: /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/mapfiles/wsclean_image_full1-image.fits.mapfile 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Writing data map file: /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/mapfiles/wsclean_image_full1-model.fits.mapfile 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: recipe executable_args completed 2016-11-06 16:38:49 INFO facetimage_facet_patch_539: Beginning step create_imagebase_map1 2016-11-06 16:38:49 INFO facetimage_facet_patch_539: Beginning step adjust_wsclean_mapfile1 2016-11-06 16:38:49 INFO facetimage_facet_patch_539: Beginning step mask 2016-11-06 16:38:49 INFO facetimage_facet_patch_539: Running task: make_clean_mask 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: recipe executable_args started 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: Starting /usr/local/lib/python2.7/dist-packages/FACTOR-1.0-py2.7.egg/factor/scripts/make_clean_mask.py run 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Pipeline start time: 2016-11-06T20:55:48 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: Limiting to 24 simultaneous jobs/node 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Job dispatcher at 127.0.1.1:46391 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: ** Remote method is local 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: Waiting for compute threads... 2016-11-06 16:38:49 INFO facetimage_facet_patch_539.executable_args: Subprocess starting: /bin/sh -c python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 46391 (['/bin/sh', '-c', 'python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 46391']) 2016-11-06 16:38:49 WARNING facetimage_facet_patch_539.executable_args: /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module! 2016-11-06 16:38:49 DEBUG facetimage_facet_patch_539.executable_args: Request for job 0 from ('127.0.0.1', 34032) 2016-11-06 16:38:49 DEBUG node.lofar.python_plugin: infile = /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits 2016-11-06 16:38:49 DEBUG node.lofar.python_plugin: executable = /usr/local/lib/python2.7/dist-packages/FACTOR-1.0-py2.7.egg/factor/scripts/make_clean_mask.py 2016-11-06 16:38:49 DEBUG node.lofar.python_plugin: working directory = /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539 2016-11-06 16:38:49 DEBUG node.lofar.python_plugin: arguments = ['/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits', '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.mask'] 2016-11-06 16:38:49 DEBUG node.lofar.python_plugin: arg dictionary = {'threshpix': '5', 'atrous_jmax': '3', 'adaptive_rmsbox': 'True', 'img_format': 'fits', 'rmsbox_bright': '(70,10)', 'threshisl': '3', 'vertices_file': '/p3600/rvweeren/P214_52_full/state/facet_patch_539_save.pkl', 'region_file': '[]', 'rmsbox': '(300,100)', 'atrous_do': 'True'} 2016-11-06 16:38:49 DEBUG node.lofar.python_plugin: environment = {'SHLIB_PATH': '/home/rvweeren/software/root/lib:', 'LOFARDATAROOT': '/opt/lofar/data', 'OMP_NUM_THREADS': '8', 'PYTHONPATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib/python:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/::/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp', 'LIBPATH': '/home/rvweeren/software/root/lib:', 'DYLD_LIBRARY_PATH': '/home/rvweeren/software/root/lib:', 'LOFARROOT': '/home/rvweeren/software/lofaroct2016/opt/LofIm2', 'MANPATH': '/home/rvweeren/software/root/man:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/man:/usr/local/man:/usr/local/share/man:/usr/share/man', 'PATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/sbin:/home/rvweeren/software/lofaroct2016/opt/LofIm2/bin:/home/rvweeren/software/root/bin:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools', 'LD_LIBRARY_PATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/lib:3456LD_LIBRARY_PATH:/usr/local/lib'} 2016-11-06 16:38:49 INFO node.lofar.python_plugin: Processing /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits 2016-11-06 16:39:25 WARNING facetimage_facet_patch_539.executable_args: --> Opened '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits.blanked' 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: Image size .............................. : (3750, 3750) pixels 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: Number of channels ...................... : 1 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: Number of Stokes parameters ............. : 1 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC01_01 = 1.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC02_01 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC03_01 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC04_01 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC01_02 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC02_02 = 1.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC03_02 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC04_02 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC01_03 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC02_03 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC03_03 = 1.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC04_03 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC01_04 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC02_04 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC03_04 = 0.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: WARNING: FITSFixedWarning: PC04_04 = 1.000000000000E+00 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: indices in parameterized keywords must not have leading zeroes. [astropy.wcs.wcs] 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: Beam shape (major, minor, pos angle) .... : (8.70288e-04, 8.70288e-04, 90.0) degrees 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: Frequency of image ...................... : 142.870 MHz 2016-11-06 16:39:26 WARNING facetimage_facet_patch_539.executable_args: Number of blank pixels .................. : 10531988 (74.9%) 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: Flux from sum of (non-blank) pixels ..... : -0.055 Jy 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: --> Calculating background rms and mean images 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: --> Using adaptive scaling of rms_box 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofar/bdsm/rmsimage.py:102: RuntimeWarning: invalid value encountered in greater_equal 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: act_pixels = (image-cmean)/threshold >= crms 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofar/bdsm/rmsimage.py:124: RuntimeWarning: invalid value encountered in greater_equal 2016-11-06 16:39:28 WARNING facetimage_facet_patch_539.executable_args: act_pixels = (image-cmean)/threshold >= crms 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofar/bdsm/rmsimage.py:431: RuntimeWarning: invalid value encountered in less 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: if N.any(rms < 0.0): 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: Using user-specified rms_box ............ : (70, 10) pixels (small scale) 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: Using user-specified rms_box ............ : (300, 100) pixels (large scale) 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: Number of sources using small scale ..... : 2 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: --> Using 2D map for background rms 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: --> Using constant background mean 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: Min/max values of background rms map .... : (0.00021, 0.00477) Jy/beam 2016-11-06 16:40:14 WARNING facetimage_facet_patch_539.executable_args: Value of background mean ................ : 0.0 Jy/beam 2016-11-06 16:40:15 WARNING facetimage_facet_patch_539.executable_args: Minimum number of pixels per island ..... : 6 2016-11-06 16:40:15 WARNING facetimage_facet_patch_539.executable_args: /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofar/bdsm/islands.py:164: RuntimeWarning: invalid value encountered in greater_equal 2016-11-06 16:40:15 WARNING facetimage_facet_patch_539.executable_args: act_pixels = (image-mean)/thresh_isl >= rms 2016-11-06 16:40:15 WARNING facetimage_facet_patch_539.executable_args: Number of islands found ................. : 108 2016-11-06 16:40:37 WARNING facetimage_facet_patch_539.executable_args: Total number of Gaussians fit to image .. : 62 2016-11-06 16:40:37 WARNING facetimage_facet_patch_539.executable_args: Total flux density in model ............. : 1.491 Jy 2016-11-06 16:40:39 WARNING facetimage_facet_patch_539.executable_args: 2016-11-06 16:40:39 WARNING facetimage_facet_patch_539.executable_args: --> Wavelet scale #1 2016-11-06 16:40:43 WARNING facetimage_facet_patch_539.executable_args: --> Calculating background rms and mean images 2016-11-06 16:40:43 WARNING facetimage_facet_patch_539.executable_args: --> Using adaptive scaling of rms_box 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: Using user-specified rms_box ............ : (70, 23) pixels (small scale) 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: Using user-specified rms_box ............ : (300, 100) pixels (large scale) 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: Number of sources using small scale ..... : 2 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: --> Using 2D map for background rms 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: --> Using constant background mean 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: Min/max values of background rms map .... : (0.00018, 0.00325) Jy/beam 2016-11-06 16:41:07 WARNING facetimage_facet_patch_539.executable_args: Value of background mean ................ : 0.0 Jy/beam 2016-11-06 16:41:08 WARNING facetimage_facet_patch_539.executable_args: Number of islands found ................. : 85 2016-11-06 16:42:45 ERROR node.lofar.python_plugin: each index entry must be either a slice, an integer, Ellipsis, or newaxis 2016-11-06 16:42:45 INFO node.lofar.python_plugin: Total time 235.5773s; user time: 425.7650s; system time: 12.5130s 2016-11-06 16:42:45 DEBUG node.lofar.python_plugin: Start time was 1478468329.7702s; end time was 1478468565.3477s 2016-11-06 16:42:45 DEBUG facetimage_facet_patch_539.executable_args: Results for job 0 submitted by ('127.0.0.1', 34034) 2016-11-06 16:42:46 DEBUG facetimage_facet_patch_539.executable_args: 2016-11-06 16:42:46 WARNING facetimage_facet_patch_539.executable_args: 2016-11-06 16:42:46 INFO facetimage_facet_patch_539.executable_args: Subprocess completed with exit status 1: /bin/sh -c python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 46391 2016-11-06 16:42:46 ERROR facetimage_facet_patch_539.executable_args: Remote process python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py ['/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits', '/usr/local/lib/python2.7/dist-packages/FACTOR-1.0-py2.7.egg/factor/scripts/make_clean_mask.py', ['/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits', '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.mask'], {'threshpix': '5', 'atrous_jmax': '3', 'adaptive_rmsbox': 'True', 'atrous_do': 'True', 'rmsbox_bright': '(70,10)', 'threshisl': '3', 'vertices_file': '/p3600/rvweeren/P214_52_full/state/facet_patch_539_save.pkl', 'region_file': '[]', 'rmsbox': '(300,100)', 'img_format': 'fits'}, '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'SHLIB_PATH': '/home/rvweeren/software/root/lib:', 'LOFARDATAROOT': '/opt/lofar/data', 'OMP_NUM_THREADS': '8', 'PYTHONPATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib/python:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/::/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp', 'LIBPATH': '/home/rvweeren/software/root/lib:', 'DYLD_LIBRARY_PATH': '/home/rvweeren/software/root/lib:', 'LOFARROOT': '/home/rvweeren/software/lofaroct2016/opt/LofIm2', 'MANPATH': '/home/rvweeren/software/root/man:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/man:/usr/local/man:/usr/local/share/man:/usr/share/man', 'PATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/sbin:/home/rvweeren/software/lofaroct2016/opt/LofIm2/bin:/home/rvweeren/software/root/bin:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools', 'LD_LIBRARY_PATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/lib:3456LD_LIBRARY_PATH:/usr/local/lib'}] failed on localhost 2016-11-06 16:42:46 DEBUG facetimage_facet_patch_539.executable_args: compute.dispatch results job 0: job_duration: 236.546439171, returncode: 1 2016-11-06 16:42:47 DEBUG facetimage_facet_patch_539.executable_args: Adding node_logging_information 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out! 2016-11-06 16:42:47 WARNING facetimage_facet_patch_539.executable_args: Note: recipe outputs are not complete 2016-11-06 16:42:47 WARNING facetimage_facet_patch_539.executable_args: recipe executable_args completed with errors 2016-11-06 16:42:47 WARNING facetimage_facet_patch_539: make_clean_mask reports failure (using executable_args recipe) 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: Failed pipeline run: facet_patch_539 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: Detailed exception information: 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'> 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: make_clean_mask failed 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: 2016-11-06 16:42:47 ERROR facetimage_facet_patch_539: LOFAR Pipeline finished unsuccesfully. 2016-11-06 16:42:47 WARNING facetimage_facet_patch_539: recipe facetimage_facet_patch_539 completed with errors

darafferty commented 8 years ago

Hmm. This is a new one to me -- the problem might be related to recent changes I made to the image-blanking script that is run before masking. Can you post the image ('/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits.blanked') somewhere so I can try it?

rvweeren commented 8 years ago

https://www.dropbox.com/s/uu2h2qdqqi5cfqv/David.tar.gz?dl=0

darafferty commented 8 years ago

Strangely, I can't reproduce this problem when running PyBDSM by hand with the same settings. However, I get a somewhat lower number of islands (97 vs. 108) and many more Gaussians fit (213 vs. 62) in the non-wavelet stage. I also don't get the warnings about invalid values (indicating, I think, that the rms image has NaNs in it). Perhaps there are some differences in software versions? Reinout, which version of the LOFAR trunk and astropy do you have?

rvweeren commented 8 years ago

-astropy version 1.1.2 -trunk release from about a 10 days ago (how do I get the release number?)

darafferty commented 8 years ago

I have (effectively) the same versions, so it must be something else. I did manage to reproduce the number of islands and Gaussians (I had the thresholds wrong), but not the error. However, I tweaked the masking script a bit (including the blanking part), so it's possible the error will no longer occur. Can you update Factor and try it again?

rvweeren commented 8 years ago

Yes will do, but I will wait until it finishes re-imaging the other facets (bit afraid killing it now...)

rvweeren commented 7 years ago

I restarted (updated to the latest version of today), but got the error below (I did not clean up the acetimage/facet_patch_539 dir.)

Should rm -rf facetimage/facet_patch_539 and try again ?

2016-11-10 22:21:04 INFO node.lofar.python_plugin: Processing /p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits 2016-11-10 22:21:16 ERROR node.lofar.python_plugin: [Errno 21] Is a directory: '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits.blanked' 2016-11-10 22:21:16 INFO node.lofar.python_plugin: Total time 12.1924s; user time: 10.4312s; system time: 1.7630s 2016-11-10 22:21:16 DEBUG node.lofar.python_plugin: Start time was 1478834464.5927s; end time was 1478834476.7852s 2016-11-10 22:21:16 DEBUG facetimage_facet_patch_539.executable_args: Results for job 0 submitted by ('127.0.0.1', 32827) 2016-11-10 22:21:17 DEBUG facetimage_facet_patch_539.executable_args: 2016-11-10 22:21:17 WARNING facetimage_facet_patch_539.executable_args: 2016-11-10 22:21:17 INFO facetimage_facet_patch_539.executable_args: Subprocess completed with exit status 1: /bin/sh -c python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 127.0.1.1 36261 2016-11-10 22:21:17 ERROR facetimage_facet_patch_539.executable_args: Remote process python /home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py ['/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits', '/usr/local/lib/python2.7/dist-packages/FACTOR-1.0-py2.7.egg/factor/scripts/make_clean_mask.py', ['/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits', '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539/L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.mask'], {'threshpix': '5', 'atrous_jmax': '3', 'adaptive_rmsbox': 'True', 'atrous_do': 'True', 'rmsbox_bright': '(70,10)', 'threshisl': '3', 'vertices_file': '/p3600/rvweeren/P214_52_full/state/facet_patch_539_save.pkl', 'region_file': '[]', 'rmsbox': '(300,100)', 'img_format': 'fits'}, '/p3600/rvweeren/P214_52_full/results/facetimage/facet_patch_539', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'SHLIB_PATH': '/home/rvweeren/software/root/lib:', 'LOFARDATAROOT': '/opt/lofar/data', 'OMP_NUM_THREADS': '8', 'PYTHONPATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/lib/python2.7/site-packages:/home/rvweeren/software/root/lib:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib/python:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/lib:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/:/home/rvweeren/software/wcsaxes/build/lib/python/:/home/rvweeren/software/pyvo/build/lib/python/::/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp:/home/rvweeren/scripts/pp', 'LIBPATH': '/home/rvweeren/software/root/lib:', 'DYLD_LIBRARY_PATH': '/home/rvweeren/software/root/lib:', 'LOFARROOT': '/home/rvweeren/software/lofaroct2016/opt/LofIm2', 'MANPATH': '/home/rvweeren/software/root/man:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/man:/usr/local/man:/usr/local/share/man:/usr/share/man', 'PATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/sbin:/home/rvweeren/software/lofaroct2016/opt/LofIm2/bin:/home/rvweeren/software/root/bin:/home/rvweeren/software/heasoft-6.16/x86_64-unknown-linux-gnu-libc2.19-0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools:/home/rvweeren/software/casapy/casa-release-4.5.2-el6:/home/rvweeren/software/wsclean-code/wsclean/build/:/home/rvweeren/software/ds9/:/home/rvweeren/software/UDR/src/:/home/rvweeren/software/losoto:/home/rvweeren/software/losoto/tools', 'LD_LIBRARY_PATH': '/home/rvweeren/software/lofaroct2016/opt/LofIm2/lib:3456LD_LIBRARY_PATH:/usr/local/lib'}] failed on localhost 2016-11-10 22:21:17 DEBUG facetimage_facet_patch_539.executable_args: compute.dispatch results job 0: job_duration: 13.3802978992, returncode: 1 2016-11-10 22:21:18 DEBUG facetimage_facet_patch_539.executable_args: Adding node_logging_information 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out! 2016-11-10 22:21:18 WARNING facetimage_facet_patch_539.executable_args: Note: recipe outputs are not complete 2016-11-10 22:21:18 WARNING facetimage_facet_patch_539.executable_args: recipe executable_args completed with errors 2016-11-10 22:21:18 WARNING facetimage_facet_patch_539: make_clean_mask reports failure (using executable_args recipe) 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: Failed pipeline run: facet_patch_539 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: Detailed exception information: 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'> 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: make_clean_mask failed 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: 2016-11-10 22:21:18 ERROR facetimage_facet_patch_539: LOFAR Pipeline finished unsuccesfully. 2016-11-10 22:21:18 WARNING facetimage_facet_patch_539: recipe facetimage_facet_patch_539 completed with errors

darafferty commented 7 years ago

Yes, you should delete facetimage/facet_patch_539 as you say (I switched from using casa images to fits images, so it does not work with the previously made image L343226_SBgr046-10_uv.dppp.pre-cal_chunk9.wsclean_image_full1-MFS-image.fits.blanked)

rvweeren commented 7 years ago

Ok, now it does not crash and completes successfully.

For reference: this entire factor run took 9-10 days (240 SB, 24 core single machine).