facet crashed two times

[manuorru]

Hi David, I am running factor on cep3 using the last version. A very easy facet crashed twice in tw different steps. Once 16/81 steps and now 70/81 steps. The images look pretty fine so I am wandering what could this be?

Any idea? Did you changed something?

2016-10-27 12:14:23,896 - DEBUG - factor:facet_patch_279 - Target bandwidth for facet imaging is 0.159519805908 MHz

2016-10-27 12:14:23,896 - DEBUG - factor:facet_patch_279 - Using averaging steps of 2 channels and 1 time slots for facet imaging 2016-10-27 12:14:24,453 - DEBUG - factor:facet_patch_279 - Total flux density of calibrator: 0.527926 Jy 2016-10-27 12:14:24,453 - DEBUG - factor:facet_patch_279 - Peak flux density of calibrator: 0.233751232685 Jy/beam 2016-10-27 12:14:24,453 - DEBUG - factor:facet_patch_279 - Effective flux density of calibrator: 0.402486462075 Jy 2016-10-27 12:14:24,453 - DEBUG - factor:facet_patch_279 - Using solution intervals of 2 (fast) and 120 (slow) time slots 2016-10-27 12:14:27,071 - INFO - factor:scheduler - <-- Operation facetselfcal started (direction: facet_patch_279) 2016-10-27 13:14:59,183 - ERROR - factor:scheduler - Operation facetselfcal failed due to an error (direction: facet_patch_279) 2016-10-27 13:14:59,404 - DEBUG - factor:scheduler - Time for operation: 1:00:34.947010 2016-10-27 13:14:59,405 - ERROR - factor:scheduler - One or more operations failed due to an error. Exiting...

[AHorneffer]

The first facet that I ran with the wsclean branch version of factor yesterday also chrashed. It looks like the step "apply_dir_dep" didn't actually do anything, and thus the following step failed.

The last part of the log is quoted here, and the full logfile is currently uploading on dropbox.

2016-10-27 23:18:37 INFO facetselfcal_facet_patch_276: Beginning step expand_sourcedb_all_facet_sources 2016-10-27 23:18:37 INFO facetselfcal_facet_patch_276: Beginning step select_imaging_bands 2016-10-27 23:18:37 INFO facetselfcal_facet_patch_276: Beginning step select_sourcedb_all_facet_sources 2016-10-27 23:18:37 INFO facetselfcal_facet_patch_276: Beginning step select_parmdb_map 2016-10-27 23:18:37 INFO facetselfcal_facet_patch_276: Beginning step select_merged_parmdb_map 2016-10-27 23:18:37 INFO facetselfcal_facet_patch_276: Beginning step apply_dir_dep 2016-10-27 23:18:38 INFO facetselfcal_facet_patch_276: Running task: dppp 2016-10-27 23:18:38 INFO facetselfcal_facet_patch_276.executable_args: recipe executable_args started 2016-10-27 23:18:38 INFO facetselfcal_facet_patch_276.executable_args: Starting /opt/soft/lofar-stuff/bin/DPPP run 2016-10-27 23:18:38 DEBUG facetselfcal_facet_patch_276.executable_args: Pipeline start time: 2016-10-27T15:25:01 2016-10-27 23:18:38 INFO facetselfcal_facet_patch_276.executable_args: Limiting to 5 simultaneous jobs/node 2016-10-27 23:18:38 DEBUG facetselfcal_facet_patch_276.executable_args: Job dispatcher at 134.104.22.26:58748 2016-10-27 23:18:38 INFO facetselfcal_facet_patch_276.executable_args: Waiting for compute threads... 2016-10-27 23:18:39 DEBUG facetselfcal_facet_patch_276.executable_args: Adding node_logging_information 2016-10-27 23:18:39 DEBUG facetselfcal_facet_patch_276.executable_args: Writing data map file: /media/scratch/test/horneff/Factor-IC324/work-first/results/facetselfcal/facet_patch_276/mapfiles/apply_dir_dep.mapfile 2016-10-27 23:18:39 INFO facetselfcal_facet_patch_276.executable_args: recipe executable_args completed 2016-10-27 23:18:39 INFO facetselfcal_facet_patch_276: Beginning step create_compressed_mapfile5 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: Failed pipeline run: facet_patch_276 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: Detailed exception information: 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: <type 'exceptions.IndexError'> 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: list index out of range 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: 2016-10-27 23:18:39 ERROR facetselfcal_facet_patch_276: LOFAR Pipeline finished unsuccesfully. 2016-10-27 23:18:42 WARNING facetselfcal_facet_patch_276: recipe facetselfcal_facet_patch_276 completed with errors

[AHorneffer]

O.K. the "imaging_bands.mapfile" is empty, and thus all the following mapfiles are also empty...

....

Ah, you look at file-names! But prefactor 2 has a different naming convention than prefactor 1! So the plugin thinks that all my data is in the single frequency band "SB000"!

[darafferty]

Oh yes -- I meant the name matching to be a temporary change (to speed up testing). Removing it now...

[AHorneffer]

Well, there is a good reason not to open all MSs in a plugin.

Do you already have a plan to avoid that, or shall I look into that?

[AHorneffer]

Manu: If yours is the same problem as mine, then you need to remove the last step from the statefile to re-run Factor successfully. (Or you can reset the full operation.)

[darafferty]

By the way, the testing I mention above was to try to understand a very strange issue with mapfiles getting "mixed" when we run two "selectMiddleFreq" plugins, one after the other. The mapfile from the second run would include the files from the first run as well. I eventually fixed it (I hope) by explicitly unsetting various attributes of the datamaps.

I don't have a good idea to avoid opening all the MSs. Maybe pass the frequencies (which we know already of course) as a list to the plugin?

[manuorru]

Hi both, my error seems to be different it seems that the mask can not be made: I am going to post the last image that seems to be good enough I don't understand why this should prevent it.

2016-10-27 13:14:33 WARNING facetselfcal_facet_patch_279.executable_args: Image size .............................. : (5040, 5040) pixels 2016-10-27 13:14:33 WARNING facetselfcal_facet_patch_279.executable_args: Number of channels ...................... : 1 2016-10-27 13:14:33 WARNING facetselfcal_facet_patch_279.executable_args: Number of Stokes parameters ............. : 1 2016-10-27 13:14:33 WARNING facetselfcal_facet_patch_279.executable_args: Beam shape (major, minor, pos angle) .... : (2.14425e-03, 1.52491e-03, -61.3) degrees 2016-10-27 13:14:33 WARNING facetselfcal_facet_patch_279.executable_args: Frequency of image ...................... : 145.018 MHz 2016-10-27 13:14:33 WARNING facetselfcal_facet_patch_279.executable_args: Number of blank pixels .................. : 21860989 (86.1%) 2016-10-27 13:14:34 WARNING facetselfcal_facet_patch_279.executable_args: Flux from sum of (non-blank) pixels ..... : 1.299 Jy 2016-10-27 13:14:34 WARNING facetselfcal_facet_patch_279.executable_args: --> Calculating background rms and mean images 2016-10-27 13:14:34 WARNING facetselfcal_facet_patch_279.executable_args: --> Using adaptive scaling of rms_box 2016-10-27 13:14:34 WARNING facetselfcal_facet_patch_279.executable_args: /opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofar/bdsm/rmsimage.py:102: RuntimeWarning: invalid value encountered in greater_equal 2016-10-27 13:14:34 WARNING facetselfcal_facet_patch_279.executable_args: act_pixels = (image-cmean)/threshold >= crms 2016-10-27 13:14:35 WARNING facetselfcal_facet_patch_279.executable_args: /opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofar/bdsm/rmsimage.py:124: RuntimeWarning: invalid value encountered in greater_equal 2016-10-27 13:14:35 WARNING facetselfcal_facet_patch_279.executable_args: act_pixels = (image-cmean)/threshold >= crms 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: /opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofar/bdsm/rmsimage.py:431: RuntimeWarning: invalid value encountered in less 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: if N.any(rms < 0.0): 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: Using user-specified rms_box ............ : (80, 20) pixels (small scale) 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: Using user-specified rms_box ............ : (80, 20) pixels (large scale) 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: --> Using 2D map for background rms 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: --> Using constant background mean 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: Min/max values of background rms map .... : (0.00018, 0.00371) Jy/beam 2016-10-27 13:14:55 WARNING facetselfcal_facet_patch_279.executable_args: Value of background mean ................ : 0.0 Jy/beam 2016-10-27 13:14:56 WARNING facetselfcal_facet_patch_279.executable_args: Minimum number of pixels per island ..... : 7 2016-10-27 13:14:56 WARNING facetselfcal_facet_patch_279.executable_args: /opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofar/bdsm/islands.py:164: RuntimeWarning: invalid value encountered in greater_equal 2016-10-27 13:14:56 WARNING facetselfcal_facet_patch_279.executable_args: act_pixels = (image-mean)/thresh_isl >= rms 2016-10-27 13:14:56 WARNING facetselfcal_facet_patch_279.executable_args: Number of islands found ................. : 0 2016-10-27 13:14:56 DEBUG facetselfcal_facet_patch_279.executable_args: No islands found. Clean mask cannot be made. 2016-10-27 13:14:56 INFO node.lof014.python_plugin: Total time 281.9732s; user time: 137.2886s; system time: 155.2737s 2016-10-27 13:14:56 DEBUG node.lof014.python_plugin: Start time was 1477573814.6988s; end time was 1477574096.6721s 2016-10-27 13:14:57 DEBUG facetselfcal_facet_patch_279.executable_args: 2016-10-27 13:14:57 WARNING facetselfcal_facet_patch_279.executable_args: 2016-10-27 13:14:57 INFO facetselfcal_facet_patch_279.executable_args: Subprocess completed with exit status 1: /bin/sh -c python /opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 10.144.5.116 51793 2016-10-27 13:14:57 ERROR facetselfcal_facet_patch_279.executable_args: Remote process python /opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py ['/data/scratch/orru/LC2_008/factor_wd_wscl/results/facetselfcal/facet_patch_279/L250378_SBgr004-10_uv.dppp.pre-cal_chunk0.wsclean_image_full-image.fits', '/home/rafferty/LOFAR/factor/factor/scripts/make_clean_mask.py', ['/data/scratch/orru/LC2_008/factor_wd_wscl/results/facetselfcal/facet_patch_279/L250378_SBgr004-10_uv.dppp.pre-cal_chunk0.wsclean_image_full-image.fits', '/data/scratch/orru/LC2_008/factor_wd_wscl/results/facetselfcal/facet_patch_279/L250378_SBgr004-10_uv.dppp.pre-cal_chunk0.wsclean_image_full-image.mask5'], {'threshpix': '20', 'adaptive_rmsbox': 'True', 'atrous_do': 'False', 'threshisl': '10', 'vertices_file': '/data/scratch/orru/LC2_008/factor_wd_wscl/state/facet_patch_279_save.pkl', 'region_file': '[]', 'rmsbox': '(80,20)', 'exclude_cal_region': 'True', 'img_format': 'fits'}, '/data/scratch/orru/LC2_008/factor_wd_wscl/results/facetselfcal/facet_patch_279', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'LOFARDATAROOT': '/opt/lofar/data', 'OMP_NUM_THREADS': '8', 'PYTHONPATH': '/home/rafferty/.local/lib/python:/home/rafferty/LOFAR/wcsaxes/lib/python/:/home/rafferty/LOFAR/LSMTool:/home/rafferty/LOFAR/factor:/opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib/python2.7/site-packages:/home/rafferty/.local/lib/python:/home/rafferty/LOFAR/wcsaxes/lib/python/:/home/rafferty/LOFAR/LSMTool:/home/rafferty/LOFAR/factor:/home/rafferty/.local/lib/python:/home/rafferty/LOFAR/wcsaxes/lib/python/:/home/rafferty/LOFAR/LSMTool:/home/rafferty/LOFAR/factor:/opt/cep/pyrap/current/lib/python2.7/site-packages:/opt/cep/lofar/external/lib/python/site-packages:/opt/cep/pythonlibs/lib/python/site-packages', 'LOFARROOT': '/opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt', 'PATH': '/home/rafferty/LOFAR/LSMTool/bin:/home/rafferty/LOFAR/factor/bin:/opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/sbin:/opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/bin:/home/rafferty/LOFAR/LSMTool/bin:/home/rafferty/LOFAR/factor/bin:/home/rafferty/LOFAR/LSMTool/bin:/home/rafferty/LOFAR/factor/bin:/opt/cep/casarest/current/bin:/opt/cep/casacore/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/cep/scripts:/opt/cep/saoimage/bin:/opt/cep/pythonlibs/bin:/opt/cep/WSClean/bin', 'LD_LIBRARY_PATH': '/home/rafferty/.local/lib:/opt/cep/LofIm/daily/Thu/lofar_build/install/gnu_opt/lib:/home/rafferty/.local/lib:/opt/cep/wcslib/lib:/opt/cep/pyrap/current/lib:/opt/cep/casarest/current/lib:/opt/cep/casacore/current/lib:.'}] failed on localhost 2016-10-27 13:14:57 DEBUG facetselfcal_facet_patch_279.executable_args: compute.dispatch results job 0: job_duration: 284.000304937, returncode: 1 2016-10-27 13:14:58 DEBUG facetselfcal_facet_patch_279.executable_args: Adding node_logging_information 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out! 2016-10-27 13:14:58 WARNING facetselfcal_facet_patch_279.executable_args: Note: recipe outputs are not complete 2016-10-27 13:14:58 WARNING facetselfcal_facet_patch_279.executable_args: recipe executable_args completed with errors 2016-10-27 13:14:58 WARNING facetselfcal_facet_patch_279: make_clean_mask reports failure (using executable_args recipe) 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: Failed pipeline run: facet_patch_279 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: Detailed exception information: 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'> 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: make_clean_mask failed 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: 2016-10-27 13:14:58 ERROR facetselfcal_facet_patch_279: LOFAR Pipeline finished unsuccesfully. 2016-10-27 13:14:59 WARNING facetselfcal_facet_patch_279: recipe facetselfcal_facet_patch_279 completed with errors

[manuorru]

[AHorneffer]

I don't have a good idea to avoid opening all the MSs. Maybe pass the frequencies (which we know already of course) as a list to the plugin?

Well, it took less than 20 seconds on my machine here, so maybe it's not so important right now.

[darafferty]

Manu, what does the*_full* image look like? The bug Andreas found will also affect that image (if you are running on more than 5 bands), and you will need to remove steps up to and including "select_imaging_bands".

[manuorru]

[manuorru]

indeed the full image looks pretty bad. It is not clear to me what I need to update? Do I need to edit the state file?

[darafferty]

Yes, you need to edit the statefile or simply reset the facet (with "runfactor your_factor.parset -r facet_patch_279")

[manuorru]

I did not find in the statefle of that facet the select_imaging_bands. I can give it a try to re-run it as I did before with -r facet_patch_279

[manuorru]

Hi David over and over again it crashed at step 70/81. Can you check the log? I don't think I had the same error Andreas had.

[manuorru]

pipeline.txt

[darafferty]

I finally tracked down the problem. It was due to time gaps not being properly propagated when the combined selfcal parmdbs were converted to gain-type solutions (and so should only affect interleaved observations). It should be fixed now on the master branch, so if you have interleaved observations, please update and restart your run from scratch

[AHorneffer]

Will that affect all interleaved observations?

[darafferty]

Yes (well, all those with time gaps between the observations, which I suppose is all of them?). Basically, the gaps were being ignored and the solutions after each gap shifted to earlier times. (Andreas, I thought you were using non-interleaved observations, since you seemingly did not have this problem.)

[AHorneffer]

Well, it didn't crash for me and the noise in the facet-images didn't look too bad - considering that it was done on only 6 of 32 bands. But I just compared the selfcal image with the full-facet image, and there is a difference. But the noise is only about a factor two higher in the full-facet image.

I'm using the dysco compression. When doing the re-start, can I re-use the compressed-chunks?

[manuorru]

I want to run it immediately so I can tell you if it fixed the issue. Sourcing your dir in cep3 will give me the updated version?

[darafferty]

Strange that it sort of worked -- but hopefully it will improve when you restart. Yes, you can re-use the chunks. Just delete the results directory and the facet*.pkl files in the state directory before restarting (you can leave the Band*.pkl files so that it will skip the check for flagged data).

[darafferty]

Yes, I updated my installation on CEP3

[AHorneffer]

Well, the observation had pretty good ionospheric conditions, so the ionospheric effects weren't that big, and the resulting images didn't look that bad. They do look bad when I compare them to the selfcal images.

[manuorru]

David, also for me it worked pretty well for the first facet. So I guess it depends on the configuration of the sources in the facet. It is running now I will let you know if you can close the issue!!

Cheers, Manu

AHorneffer notifications@github.com 11/08/16 2:19 PM >>> Well, the observation had pretty good ionospheric conditions, so the ionospheric effects weren't that big, and the resulting images didn't look that bad. They do look bad when I compare them to the selfcal images. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

NaN.

[darafferty]

Manu, I think your first facet was a patch-type facet, so it did not do the facet imaging. However, the subtract for it would still have been poor, so you should restart from scratch if you didn't do so.

[manuorru]

I see. Anyway I restarted from scratch. Thanks Manu

Sent from my iPhone

On 8 Nov 2016, at 15:46, David Rafferty notifications@github.com wrote:

Manu, I think your first facet was a patch-type facet, so it did not do the facet imaging. However, the subtract for it would still have been poor, so you should restart from scratch if you didn't do so.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[manuorru]

Now it did not crashed but the selfcal failed, but the images of the selfcal look good.

The log in the end says:

2016-11-08 16:41:41 INFO facetselfcal_facet_patch_674.executable_args: Subprocess completed with exit status 0: /bin/sh -c python /opt/cep/LofIm/daily/Tue/lofar_build/install/gnu_opt/lib/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py 0 10.144.5.116 40415 2016-11-08 16:41:41 DEBUG facetselfcal_facet_patch_674.executable_args: compute.dispatch results job 0: job_duration: 1.48083806038, returncode: 0 2016-11-08 16:41:42 DEBUG facetselfcal_facet_patch_674.executable_args: Adding node_logging_information 2016-11-08 16:41:42 DEBUG facetselfcal_facet_patch_674.executable_args: Writing data map file: /data/scratch/orru/LC2_008/factor_wd_wscl/results/facetselfcal/facet_patch_674/mapfiles/verify_subtract.mapfile 2016-11-08 16:41:42 INFO facetselfcal_facet_patch_674.executable_args: recipe executable_args completed 2016-11-08 16:41:42 INFO facetselfcal_facet_patch_674: LOFAR Pipeline finished succesfully. 2016-11-08 16:41:42 INFO facetselfcal_facet_patch_674: recipe facetselfcal_facet_patch_674 completed

And then it stops the processing. Is there still something odd?

[manuorru]

Let me know if I need to copy the all folder.

[darafferty]

The failed selfcal verification is likely due to the extended emission not being well subtracted (and hence leaving a large residual). There is no error in the log because there was no error in processing (such as a process crashing). We should probably think of a better way of doing the verification in cases like this...

For now, you can edit the mapfile in results/facetselfcal/facet_patch_674/mapfiles/verify_subtract.break.mapfile and change 'file': 'False' to 'file': 'True'. Then, restart Factor and it will now set the facet as successfully selfcaled.

[AHorneffer]

FYI: I just committed a fix to a bug in David's fix that caused the "convert_solutions_to_gain.py" to take incredible long. (Due to rounding it "detected" a gap at every second sample...) If your Factor run also gets stuck in that step, you can just kill it, upgrade the code, and re-start it.

[manuorru]

Thanks both. I am going to restart it now. Andreas is this fix already available on cep3 on David's dir?

Thanks

Manu

On 09/11/2016 11:07, AHorneffer wrote:

FYI: I just committed a fix to a bug in David's fix that caused the "convert_solutions_to_gain.py" to take incredible long. (Due to rounding it "detected" a gap at every second sample...) If your Factor run also gets stuck in that step, you can just kill it, upgrade the code, and re-start it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lofar-astron/factor/issues/147#issuecomment-259377409, or mute the thread https://github.com/notifications/unsubscribe-auth/AVW0Lb-60gdo4o58PYV5y8XXi_PNRUbEks5q8ZtSgaJpZM4Kifkw.

Dr. E. Orru' LOFAR support scientist ASTRON Oude Hoogeveensedijk 4 7991 PD Dwingeloo The Netherlands

tel: (+31) 521 - 595 761 fax : (+31) 521 - 595 101 email: orru@astron.nl

[darafferty]

Thanks, Andreas! Yes, Factor on CEP3 has been updated

[manuorru]

ok thanks I am going to restart.

[manuorru]

Now is running. I will let you know how it goes for the next facet after that we can close the issue ;-)

[manuorru]

I am almost emotional to confirm that the third facet run successfully. Well done David! You can close the issue.

[AHorneffer]

Hmmm... the first facet completed, but there is still a difference between the last selfcal image and the full facet image:

[darafferty]

I think I've found the issue: for some reason, the gaps are being ignored in the slow gain solutions in the conversion script (so the amplitudes solutions get shifted in time, resulting in the errors in the image). Not sure yet why the fast phases are fine and the slow gains aren't. Anyway, I am working on a fix now...

[manuorru]

Great. I will finish my copy anyway in case you will need an interleaved dataset to test things.

[manuorru]

Hi David any news about the fix?

[darafferty]

Getting the interpolated values from the slow and fast parmdbs separately (rather than from the merged one) seems to give the correct results (left is from merged, right is from separate):

I am working now to update the pipeline parsets to pass the separate parmdbs.

[darafferty]

This issue should hopefully be fixed now (on CEP3 too). Manu and Andreas, can you try your runs again?

[AHorneffer]

I re-started my run.

[AHorneffer]

I still see similar artefacts in the full facet image. So I did a factor run on only 80 SBs (8 bands of 10 SBs) so that the difference in data is not much different between the selfcal images (which AFAIK are done on the full bandwidth) and the full facet images (which are only done on a fraction of the bandwidth. Again there are artefacts in the full facet image, but the RMS "noise" is actually lower in the full facet image than in the selfcal image. So I think it is possible that the artefacts are due to the cleaning.

[manuorru]

Hi both,

sorry for my absence I was off for few days. Andreas if the artefacts are due to the cleaning then on the same amount of data we should be able to see them also on the long runs. David can you confirm this?

Cheers Manu

Dr. E. Orru

LOFAR support scientist ASTRON
Oude Hoogeveensedijk 4
7991 PD Dwingeloo
The Netherlands

tel: (+31) 521 - 595 761 fax : (+31) 521 - 595 101 email: orru@astron.nl

On 11 Nov 2016, at 14:39, AHorneffer notifications@github.com wrote:

I still see similar artefacts in the full facet image. So I did a factor run on only 80 SBs (8 bands of 10 SBs) so that the difference in data is not much different between the selfcal images (which AFAIK are done on the full bandwidth) and the full facet images (which are only done on a fraction of the bandwidth. Again there are artefacts in the full facet image, but the RMS "noise" is actually lower in the full facet image than in the selfcal image. So I think it is possible that the artefacts are due to the cleaning.

https://cloud.githubusercontent.com/assets/900708/20216646/0c310b88-a81c-11e6-993b-65fe6598f8db.png https://cloud.githubusercontent.com/assets/900708/20216645/0c30c06a-a81c-11e6-9f42-6a3b490a11db.png — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lofar-astron/factor/issues/147#issuecomment-259959081, or mute the thread https://github.com/notifications/unsubscribe-auth/AVW0La227b0e2WAhebu8OScDmKJlcBXUks5q9HAhgaJpZM4Kifkw.

[darafferty]

The artifacts could be due to the fact that we use 6 bands spread across the full bandwidth (leaving frequency gaps). In the real facet image, we use all the bands. However, as long as the 6-band image is good enough to pick up the bright sources for subtraction, it should work fine.

[darafferty]

I just noticed in some of my (non-interleaved) parmdbs that some of the fast phase solutions are exactly 0.0. They are regularly spaced, perhaps related to the time interval used in chunking. I'm not sure where they came from, but I wonder if they could be the cause of the poorer facet images. Andreas, can you check for them as well (look in the preconverted parmdbs, as post conversion the values are no longer exactly zero)?

I've adjusted the conversion script to set these solutions to NaN, which I think will result in the corresponding time slots being flagged (right?).

[manuorru]

Hi guys, I run the first two facets and now the full looks better than the selfcal. Is that what we want? Consider that I am using only 40 SBs.

[AHorneffer]

David: I noticed that also in my data: the last time-step in a chunk has phase zero. (Well: CommonScalarPhase plus TEC, referenced to another station in parmdbplot.py) I didn't think too much of it, it is already in the original parmDBs from NDPPP.

[manuorru]

I am plotting L250378_SBgr000-10_uv.dppp.pre-cal_chunk0_125660195t_0g.merge_phase_parmdbs not sure if this is the correct one but I don't see it. Could be because David already fixed it when I run this facet? Or am I checking the wrong parmdbs