BDA: astrometrical errors due to uvw coordinate implementation

[bennahugo]

• xova version: bda-branch

Description

@o-smirnov @sjperkins @JSKenyon

The astrometrical errors due to the current UVW coordinate generation implementation are substantial. UVW errors, in first order, manifests as rotations and sheering. errors in the hour angle manifests as image space rotations about the tangent point. Frequency scaling errors measures as radial scaling errors.

To this end we use a dedicated commissioning test procedure to verify the astrometry of the averager using bda. A calibrator source is placed at offset positions across a large field of view. Before averaging, with the original UVW coordinates in hand the commanded positions (circles) and the measured positions (crosses) of the calibrator source matches closely across the entire field of view.

After applying bda with default settings and imaging at the same robust factor of 0.0 the UVW coordinates produced by geometrical mean results in substantial astrometrical errors, that do not obey a simple rotation - implying the offsets in hour angle and in radial spatial frequency are more random in nature and cannot be predicted and corrected post imaging. The net effect is a jitter in source positions across the image.

One would need to substantially rethink this bit of the averager before continuing to think about deployment. The easiest is to insert UVW coordinates as a finalization step, sequentially run with existing casacore routines (the measures package is known to be not thread safe). I've tried using fixvis to no avail - it crashes with segfault. Although this is not ideally expressed as a compute graph I don't think it is worth the development effort to reimplement these coordinate systems and leap second handling (seeing that they are not currently available in astropy/pyephem). In this case the need for accuracy far outweighs the need for absolute speed - even for moderate database sizes uvw generation using casacore/measures is not a time consuming process in the grand scheme of things.

[bennahugo]

Actually this will effect not only the BDA code path but the normal codepath as well. I will double check this

[bennahugo]

Wait scratch this post. Looking at the images now they are completely non-nonsensical. Not sure if this is because I'm dealing with a multi field ms @sjperkins

[sjperkins]

Phew! I'd still be interested in the results of your analysis on a more helpful dataset.

On Tue, 8 Sep 2020, 20:02 Benjamin Hugo, notifications@github.com wrote:

Wait scratch this post. Looking at the images now they are completely non-nonsensical. Not sure if this is because I'm dealing with a multi field ms @sjperkins https://github.com/sjperkins

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ska-sa/xova/issues/18#issuecomment-689044195, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA253ZCSKOACZGN7HAAGW6DSEZWU3ANCNFSM4RAIHUGQ .

[bennahugo]

I'm not sure what the cause is but I cannot run xova successfully. In every dataset I process the output images are slow varying ramps, indicating that the short spacings might be completely mangled. I have to move on to other work now, so we will need to postpone this till a future point.

[o-smirnov]

This is still on stevie, right @bennahugo? I'll try running there myself, to see if it's a host issue, or something with your particular setup. For the record, could you post the paths of an example pre-BDA and post-BDA MS here showing the problem.

[bennahugo]

hugo@stevie:~/widefield_pol/msdir/1579141864_sdp_l0.POLCALIBRATED.f30.ms and msdir/1579141864_sdp_l0.POLCALIBRATED.f30_averaged.ms respectively

I get crap like this:

As we have already imaged the dataset averaged on another machine with the same imager installation we already excluded the imager as the cause of the bug.

Runtime parameters

DDF.py --Parallel-NCPU 32 --Data-MS "msdir/1579141864_sdp_l0.POLCALIBRATED.f30_averaged.ms//D*//F0" --Data-ColName "DATA" --Image-NPix 128 --Image-Cell 2.0 --Output-Images dDN --Output-Cubes '' --Output-Mode Dirty --Freq-NBand 5 --Freq-NDegridBand 8 --Weight-Mode Briggs --Facets-NFacets 1 --Deconv-PeakFactor 0.2 --Parallel-NCPU 56 --Weight-ColName WEIGHT_SPECTRUM --Beam-Model None --Beam-FITSFile 'input/mattieu.feb2020.arrayavg.$(xy).$(reim).fits' --Beam-FITSLAxis X --Beam-FITSMAxis Y --Log-Boring 0 --Log-Memory 0 --Data-ChunkHours 0.75 --Cache-Reset 1 --Image-PhaseCenterRADEC 13:31:08.290000,+30:30:33.00000 --Beam-FITSLAxis X --Beam-FITSMAxis Y --Beam-FITSFeedSwap False --Beam-FeedAngle -90.0 --Output-Name "offsets_avg/f${i}" --Weight-Robust 0.0

Averaging parameters

xova bda msdir/1579141864_sdp_l0.POLCALIBRATED.f30.ms -dc DATA --decorrelation 0.99 -fov 5  -mc 16 

Installation environment

(venv3) > $ pip freeze                                                                                                                           
aiocontextvars==0.2.2
APLpy==2.0.3
appdirs==1.4.4
argon2-cffi==20.1.0
astLib==0.11.4
astro-kittens==1.4.3
astro-pyxis==1.6.1
astropy==4.0.1.post1
astropy-healpix==0.5
attrs==20.2.0
backcall==0.2.0
backports.shutil-get-terminal-size==1.0.0
bdsf==1.9.2
bleach==3.1.5
certifi==2020.6.20
cffi==1.14.2
codex-africanus @ git+https://github.com/ska-sa/codex-africanus.git@7418c6f2ef4872dacd5238fb76e861348c7b3be7
configparser==5.0.0
contextvars==2.4
cycler==0.10.0
Cython==0.29.21
dask==2.25.0
dask-ms @ git+https://github.com/ska-sa/dask-ms.git@95b1d7af1015d77f0df06fe6777f205b9331848f
-e git+git@github.com:cyriltasse/DDFacet.git@af391e7db1251661e88315cdc379873cc07b79ff#egg=DDFacet
deap==1.3.1
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
ephem==3.7.7.0
future==0.18.2
imageio==2.9.0
immutables==0.14
importlib-metadata==1.7.0
ipdb==0.13.3
ipykernel==5.3.4
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.2
Jinja2==2.11.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-core==4.6.3
kittens==0.1.2
kiwisolver==1.2.0
llvmlite==0.34.0
loguru==0.5.2
MarkupSafe==1.1.1
matplotlib==3.3.1
meqtrees-cattery==1.7.0
mistune==0.8.4
nbconvert==5.6.1
nbformat==5.0.7
networkx==2.5
nose==1.3.7
notebook==6.1.3
numba==0.51.2
numexpr==2.7.1
numpy==1.19.1
packaging==20.4
pandas==1.1.1
pandocfilters==1.4.2
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
pkg-resources==0.0.0
Polygon3==3.0.8
prettytable==0.7.2
prometheus-client==0.8.0
prompt-toolkit==3.0.7
psutil==5.7.2
ptyprocess==0.6.0
purr==1.5.0
py-cpuinfo==4.0.0
PyAVM==0.9.5
pybind11==2.5.0
pycparser==2.20
pyephem==3.7.7.0
pyFFTW==0.12.0
pyfits==3.5
Pygments==2.6.1
pylru==1.2.0
pyparsing==2.4.7
pyregion==2.0
pyrsistent==0.16.0
python-casacore==3.3.1
python-dateutil==2.8.1
pytz==2020.1
PyWavelets==1.1.1
PyYAML==5.3.1
pyzmq==19.0.2
qtconsole==4.7.7
QtPy==1.9.0
reproject==0.7.1
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
scikit-image==0.17.2
scipy==1.3.3
Send2Trash==1.5.0
Shapely==1.7.1
SharedArray==3.2.1
six==1.15.0
tables==3.6.1
terminado==0.8.3
testpath==0.4.4
tifffile==2020.9.3
toolz==0.10.0
tornado==6.0.4
traitlets==4.3.3
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
xova @ file:///home/bhugo/widefield_pol/xova
zipp==3.1.0

[sjperkins]

Note that there are around 3 changes which have gone in since the FOV change:

https://github.com/ska-sa/codex-africanus/compare/7a2bd7f0637308ca02b5b0ab9fa4022ffb388d9a...7418c6f2ef4872dacd5238fb76e861348c7b3be7#diff-84a51c171d31f55dbfc51aa18aa15f3a

1. Introduced a time_bin_secs option https://github.com/ska-sa/codex-africanus/compare/7a2bd7f0637308ca02b5b0ab9fa4022ffb388d9a...7418c6f2ef4872dacd5238fb76e861348c7b3be7#diff-84a51c171d31f55dbfc51aa18aa15f3aR186
2. Slight improvement in calculation of baseline speed https://github.com/ska-sa/codex-africanus/compare/7a2bd7f0637308ca02b5b0ab9fa4022ffb388d9a...7418c6f2ef4872dacd5238fb76e861348c7b3be7#diff-84a51c171d31f55dbfc51aa18aa15f3aR180
3. Fixing computation of EXPOSURE and WEIGHT: https://github.com/ska-sa/codex-africanus/compare/7a2bd7f0637308ca02b5b0ab9fa4022ffb388d9a...7418c6f2ef4872dacd5238fb76e861348c7b3be7#diff-9d3fdfe9a9b54d39685ee69f1c90161aR75

[sjperkins]

Hmmmm, those links are failing for me in some cases...

[sjperkins]

In any case, take a look at the last 6 or commits in the PR: https://github.com/ska-sa/codex-africanus/pull/173

[bennahugo]

I'm at:

commit b88c1a7db3a5b09ab6b2084c01cce5ca7d7b3aaf (HEAD -> bda, origin/bda)
Author: Simon Perkins <simon.perkins@gmail.com>
Date:   Thu Aug 27 13:44:38 2020 +0200

    Mention FOV is the radius

[sjperkins]

OK I think that's probably the commit Oleg was running. The mystery deepens!

[o-smirnov]

And xova commits? Anyway, it ran fine on another box, so I predict it's going to be something trivial tech detail.

[sjperkins]

OK I think that's probably the commit Oleg was running. The mystery deepens!

Sorry, I thought this was the codex branch. @bennahugo looks like you're at the tip of both the codex-africanus time-and-channel-bda and the xova bda branch.

[o-smirnov]

Good news is, I'm getting the same nonsense as you @bennahugo. I'll roll back xova and codex to whatever is working right on the other node and try again.

[o-smirnov]

Aha! This is nothing more serious than a case of bleeding edge syndrome. I rolled back to https://github.com/ska-sa/codex-africanus/commit/7a2bd7f0637308ca02 and https://github.com/ska-sa/xova/commit/b3010884528c7551e0afe298aae0b6d0b4a833f9, and my dirty is sensible again:

[sjperkins]

Aha! This is nothing more serious than a case of bleeding edge syndrome. I rolled back to ska-sa/codex-africanus@7a2bd7f and b301088, and my dirty is sensible again:

Ok I 'll take a look at the new changes to see what broke things:

1. Introduced a time_bin_secs option ska-sa/codex-africanus@7a2bd7f...7418c6fdiff-84a51c171d31f55dbfc51aa18aa15f3aR186
2. Slight improvement in calculation of baseline speed ska-sa/codex-africanus@7a2bd7f...7418c6fdiff-84a51c171d31f55dbfc51aa18aa15f3aR180
3. Fixing computation of EXPOSURE and WEIGHT: ska-sa/codex-africanus@7a2bd7f...7418c6fdiff-9d3fdfe9a9b54d39685ee69f1c90161aR75

I tested (3) and (1). Maybe (2) is the issue...

[sjperkins]

Is there a pressing need for a fix? Ideally I'd like to pick this up Wednesday next week after I've dealt with another issue.

[o-smirnov]

Not according to me. We have a working version for now.

[sjperkins]

Not according to me. We have a working version for now.

OK, just note that the EXPOSURE + WEIGHT columns will be incorrect on those commits. AFAIK DDFacet doesn't use EXPOSURE and WEIGHT_SPECTRUM should provide correct values.

I've done some preliminary work on averaging multiple visibility columns at once.

[sjperkins]

I think this is also showing the need for proper BDA test cases. I've held off for now because I've found them somewhat difficult to construct. It's probably worth taking https://github.com/ska-sa/codex-africanus/pull/210 further from Wed.

[o-smirnov]

OK, just note that the EXPOSURE + WEIGHT columns will be incorrect on those commits.

No problem, doesn't matter for DDF as you noted.

Test cases, hmmm, as the Catholic priest said to the congressman, that's a great idea, but do we have the time? :P

[bennahugo]

@sjperkins I've added the casacore code to regenerate the uvw coordinates at the time centroids based on the provided UTC times and antenna positions