LOFAR Imaging Pipeline - what is working, what is not

[AlexKurek]

I tried to process data from LOFARs 2018 School at ours Observatory machines. Below I report some results:

What is fully working: under KERNs Docker image kernsuite/prefactor:latest Prefactor 2.0.2 but ONLY if I do: 1) apt-get install -y aoflagger-dev casacore-tools lofar-dev pybdsf 2) Fix for "no module named bdsm" error. (This is no longer necessary in 2.0.3(.1)): cp make_clean_mask.py make_clean_mask.pyc /usr/lib/prefactor/scripts/ in this file I changed:

from lofar import bdsm

to:

import bdsf as bdsm

What is NOT working: under ASTRONs Docker image lofaruser/imaging-pipeline:latest ( https://hub.docker.com/r/lofaruser/imaging-pipeline/tags ): Prefactor 2.0.3(1) is not working, but to make it work we first need to:

• downgrade LOSOTO to 1.0.1
• install PP (using e.g. PIP)
• upgrade RMExtract to 0.2 (not 0.3, not master)
• downgrade WSClean to 2.4 since the "trim" parameter is removed from 2.5 ( https://sourceforge.net/p/wsclean/wiki/Changelog-2.5/ )
• pip install configparser (Im not sure if this was for 2.0.3, but somewhere it was definitely necessary - anyway the pipeline will tell you)

Prefactor 2.1 is also not working, but to make it work we first need to:

• upgrade RMExtract to 0.2 (not 0.3, not master)
• upgrade LOSOTO to master (I tested commit dfcf39e)
• possibly upgrade LSMTool to master (I tested commit 2ff3dab)
• copy the missing file "convert_fits_image_to_casa.py" from Factor to /scripts/

Versions 2.0.3 and 2.1 are now both crashing at a late stage saying the same thing:

2019-02-16 13:41:03 INFO genericpipeline: Beginning step expand_sourcedb_high_deep 2019-02-16 13:41:03 ERROR genericpipeline: *** 2019-02-16 13:41:03 ERROR genericpipeline: Failed pipeline run: Initial-Subtract-Deep 2019-02-16 13:41:03 ERROR genericpipeline: Detailed exception information: 2019-02-16 13:41:03 ERROR genericpipeline: <type 'exceptions.AttributeError'> 2019-02-16 13:41:03 ERROR genericpipeline: 'list' object has no attribute 'startswith'

I also did native LOFARs software installation using Prefactor 2.1 (missing file supplied as above) LOFAR soft. 3.2.8, LOSOTO master and RMExtract 0.2. It is also NOT working and crashing as above. I suspect that this error may be related to LOFARs software version (parmdb hanfdling, too new version) or WSClean version (I will try downgrading to 2.5)?

Best regards, Alex

[gijzelaerr]

Thanks @AlexKurek for this analysis.

I'm not sure if I can solve these problems using KERN, it looks like a fragile interaction between not properly release-managed software projects. I think the right way forward is to ring a bell at ASTRON that these software packages require more centralized coordination. @tammojan what do you think.

[AlexKurek]

I checked that version Fast of Initial Subtract is working in Prefactor 2.1 in my native LOFAR software installation, where I used versions as above and LOFAR Software 3.2.9.

The same for lofaruser/imaging-pipeline:v0.2, but first I had to: update losoto to master and:

pip install configparser tables
apt install -y cython
cp prefactor-2.1 /usr  # or somewhere else; I cannot find Prefactor in this image
pip install aplpy==1.1.1

convert_fits_image_to_casa.py - supply this missing file, for example from Factor

[AlexKurek]

I uploaded an up to date instructions and Dockerfiles for installing all dependencies of Prefactor 3 here: https://github.com/lofar-astron/prefactor/issues/266

[gijzelaerr]

Hi Alex, thanks for reporting this. I'm going to close this though, all the related packaged changed significantly in the meantime, and ASTRON doesn't have any interest in financing KERN to include prefactor packages.