No statefile is created

[duyhoang-astro]

I am using the up-to-date version of FACTOR running on 2 bands. The final error says that no statefile in the facetselfcal directory is found. There are warnings before the factor stopped, but I am not sure if these relate to the creating of statefile. Anyone knows why no statefile is created? Thanks.

################# INFO - factor:parset - Working on 2 input files. INFO - factor - Setting up cluster/node parameters... INFO - factor - Using cluster setting: "local" (Single node). INFO - factor - Checking input bands... INFO - factor - Building local sky model for source avoidance and DDE calibrator selection (if desired)... WARNING: Couldn't write lextab module u'angle_lextab'. [Errno 13] Permission denied: u'/software/rhel7/lib64/python2.7/site-packages/astropy/coordinates/angle_lextab.py' WARNING: Couldn't create u'angle_parsetab'. [Errno 13] Permission denied: u'/software/rhel7/lib64/python2.7/site-packages/astropy/coordinates/angle_parsetab.py' INFO - factor:directions - Found 3066 sources through thresholding INFO - factor - Setting up directions... INFO - factor:directions - Reading directions file: /home/hoang/para/h112/factor/a_2bands/factor_directions.txt INFO - factor:directions - Adjusting facets to avoid sources... /home/hoang/para/h34/soft/LSMTool/lsmtool/skymodel.py:1438: FutureWarning: np.average currently does not preserve subclasses, but will do so in the future to match the behavior of most other numpy functions such as np.mean. In particular, this means calls which returned a scalar may return a 0-d subclass object instead. return np.average(c, axis=0) INFO - factor:directions - Including target (08h30m45.2, +65d52m55.000) in facet adjustment /home/hoang/lofar/pipelines/factor/factor_feb0917/factor/directions.py:808: RuntimeWarning: invalid value encountered in sqrt min_sizes = [fwhm*min(10.0, max(2.0, np.sqrt(flux_jy/0.01))) for flux_jy in fluxes_jy] INFO - factor - Self calibrating 1 direction(s) in Group 1 QPID support NOT enabled! Will NOT connect to any broker, and messages will be lost! INFO - factor:scheduler - <-- Operation facetselfcal started (direction: facet_patch_886) ERROR - factor:scheduler - Operation facetselfcal failed due to an error (direction: facet_patch_886) Traceback (most recent call last): File "/home/hoang/lofar/pipelines/factor/factor_feb0917/bin/runfactor", line 70, in reset_operations) File "/home/hoang/lofar/pipelines/factor/factor_feb0917/factor/process.py", line 179, in run scheduler.run(ops) File "/home/hoang/lofar/pipelines/factor/factor_feb0917/factor/lib/scheduler.py", line 302, in run op.cleanup() File "/home/hoang/lofar/pipelines/factor/factor_feb0917/factor/lib/operation.py", line 420, in cleanup steptypes = self.get_steptypes() File "/home/hoang/lofar/pipelines/factor/factor_feb0917/factor/lib/operation.py", line 368, in get_steptypes current_state = pickle.load(open(statefile, 'rb')) IOError: [Errno 2] No such file or directory: '/home/hoang/para/h112/factor/a_2bands/results/facetselfcal/facet_patch_886/statefile'

[twshimwell]

Hey Duy, I think this is an issue on our cluster since it was upgrade to redhat7... not sure what we have wrong but I suspect if you look in the logs that were produced you will see something like

2017-02-01 18:40:13 DEBUG facetselfcal_s1: Pipeline start time: 2017-02-01T17:40:13 2017-02-01 18:40:13 INFO facetselfcal_s1: LOFAR Pipeline (facetselfcal_s1) starting. 2017-02-01 18:40:13 INFO facetselfcal_s1: SASID = , MOMID = , Feedback method = None NYI: validate_steps 2017-02-01 18:40:14 ERROR facetselfcal_s1: 2017-02-01 18:40:14 ERROR facetselfcal_s1: Failed pipeline run: s1 2017-02-01 18:40:14 ERROR facetselfcal_s1: Detailed exception information: 2017-02-01 18:40:14 ERROR facetselfcal_s1: <type 'exceptions.TypeError'> 2017-02-01 18:40:14 ERROR facetselfcal_s1: No Python class registered for C++ class LOFAR::PyParameterSet 2017-02-01 18:40:14 ERROR facetselfcal_s1: 2017-02-01 18:40:14 ERROR facetselfcal_s1: LOFAR Pipeline finished unsuccesfully. 2017-02-01 18:40:14 WARNING facetselfcal_s1: recipe facetselfcal_s1 completed with errors

[duyhoang-astro]

Oh, yes. I see the same type of error with the C++ class.

2017-02-09 15:18:26 DEBUG facetselfcal_facet_patch_886: Pipeline start time: 2017-02-09T14:18:26 2017-02-09 15:18:26 INFO facetselfcal_facet_patch_886: LOFAR Pipeline (facetselfcal_facet_patch_886) starting. 2017-02-09 15:18:26 INFO facetselfcal_facet_patch_886: SASID = , MOMID = , Feedback method = None 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: Failed pipeline run: facet_patch_886 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: Detailed exception information: 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: <type 'exceptions.TypeError'> 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: No Python class registered for C++ class LOFAR::PyParameterSet 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: 2017-02-09 15:18:27 ERROR facetselfcal_facet_patch_886: LOFAR Pipeline finished unsuccesfully. 2017-02-09 15:18:27 WARNING facetselfcal_facet_patch_886: recipe facetselfcal_facet_patch_886 completed with errors

[AHorneffer]

Well, I guess the real "bug" is that Factor doesn't handle it gracefully if an operation doesn't generate a statefile, which happens when it fails during or before its first "real" step.

But the real difference would only be that the output looks a bit different.

[twshimwell]

I suspect we can close this issue as its not really a factor problem. If anyone has seen this before and has any ideas for how to fix it though we would be very grateful over here in Leiden.