wgridder seems to assume a negated `w` coordinate

[Joshuaalbert]

The phase in the RIME is:

phase = (-2j * pi / lambda * (l * u + m * v + (n - 1) * w)

but I see wgridder's implemented with negated w

phase = (-2j * pi / lambda * (l * u + m * v - (n - 1) * w)

Is this intended, and if so why?

[mreineck]

I'm suggesting the addition of flip_u and flip_w, because this is how I'll handle it internally anyway (at last for the time being); it is the least invasive way of allowing everything that's being discussed (in combination with the array stride changes).

In principle the original interface, augmented by flip_u and flip_w allows you to build the ideal gridder interface in pure Python without performance penalty, calling the current wgridder (with its admittedly clunky interface) as a workhorse in the background. In my eyes, this has several advantages:

• evolution and unavoidable tweaks to the interface can be done in Python, without recompilation and without involving myself fiddling with tedious C++ code in every step (and potentially making a new release)
• the current enhancement of the wgridder interface is backwards compatible, so there is no needfor new function names ... only the C++ calls in wsclean need very slight adjustment.
• once the new interface has converged and everyone is happy, I can adopt it into ducc0.wgridder, and potentially adjust and simplify my C++ implementation.

Could this be an acceptable way forward?

[aroffringa]

Personally I'd prefer the boolean flip parameters. casa or physical is not common terminology, before this discussion I had no idea what it meant (and 'casa' is still unclear to me).

[Joshuaalbert]

Re: casa/physical pair. It doesn't matter what the words are. Codex Africanus uses casa/fourier. TMS book uses "physical". Point is it doesn't matter as long as documentation is clear. Everyone knows you have to choose a convention. Debating the point just cost our time in my opinion with little gain.

@mreineck ultimately the choice is yours how you update wgridder. I've opened this issue because of confusion in how it was implemented and we've found that counter bugs were used in some software to deal with it. My preference is explicit conformance to DFT supporting both conventions and memory layouts as per proposal. I suppose the maiin Objective has been accomplished which was to identify the issue so that even the current interface will work. I would argue don't even update wgridder unless there is a clear improvement. I dont see adding more flip args as a benefit.

[landmanbester]

It is true that the boolean flip parameters provide the most flexibility from an external API but wouldn't you still have to specify the sign of center_x and center_y relative to which parameters you choose to flip? This may get a bit confusing. I agree with @Joshuaalbert, just document the convention clearly then we can take care of the rest. I'm not completely on board with using the fits convention as the default but if you do go that route maybe go wholeheartedly and also remove the flip_v parameter so there is no ambiguity.

Codex Africanus uses casa/fourier.

Sorry, my bad. I didn't envision these kinds of ramifications

[mreineck]

I totally agree that using the current, low-level interface is confusing and error-prone. That's why I try to make experimenting with new interfaces and converging to something really useful as easy as possible. I'm sure that the design process requires >10 intermediate steps, and having me involved with doing all the adjustments on the C++ side will slow things down considerably. I'll demonstrate what I have in mind very soon, and I hope it is a good starting point towards a gridder interface that makes everyone happy.

[mreineck]

Here is an updated test script, containing @Joshuaalbert 's proposed interface. To run this, you need to update ducc to the current head of the tweak_gridder_conventions branch.

import itertools
import ducc0
import numpy as np
import pytest

# Joshua's proposed interface, emulated on top of the low-level one
def new_dirty2vis(*,
        uvw: np.ndarray,
        freqs: np.ndarray,
        dirty: np.ndarray,
        pixsize_m: float,
        pixsize_l: float,
        center_m: float,
        center_l: float,
        epsilon: float = 1e-4,
        wgt: np.ndarray = None,
        mask: np.ndarray = None,
        do_wgridding: bool = True,
        divide_by_n: bool = True,
        sigma_min: float = 1.1,
        sigma_max: float = 2.6,
        nthreads: int = 1,
        verbosity: int = 0,
        convention: str='physical',  # Literal['physical', 'casa'] = 'physical',
        dirty_ordering: str='detector'):  #Literal['detector', 'world'] = 'detector'):

    # sanity checks
    if convention != 'physical' and convention != 'casa':
        raise RuntimeError("'convention' must be 'physical' or 'casa'")
    if dirty_ordering != 'detector' and dirty_ordering != 'world':
        raise RuntimeError("'dirty_ordering' must be 'detector' or 'world'")

    flip_u, flip_v, flip_w = False, False, True

    # for Casa convention, flip uvw signs
    if convention == 'casa':
        flip_u, flip_v, flip_w = not flip_u, not flip_v, not flip_w

    # for 'detector' ordering, flip axis 0 of dirty image
    if dirty_ordering == 'detector':
        dirty = dirty[-1::-1,:]

    return ducc0.wgridder.experimental.dirty2vis(
        uvw=uvw,
        freq=freqs,
        dirty=dirty,
        wgt=wgt,
        mask=mask,
        pixsize_x=pixsize_l,
        pixsize_y=pixsize_m,
        center_x=-center_l if flip_u else center_l,
        center_y=-center_m if flip_v else center_m,
        epsilon=epsilon,
        do_wgridding=do_wgridding,
        flip_u=flip_u,
        flip_v=flip_v,
        flip_w=flip_w,
        divide_by_n=divide_by_n,
        nthreads=nthreads,
        verbosity=verbosity,
        sigma_min=sigma_min,
        sigma_max=sigma_max,
    )

def explicit_degridder(uvw, freqs, dirty, l0, dl, m0, dm, convention, dirty_ordering):
    if dirty_ordering == 'detector':
        dirty = dirty[-1::-1,:]

    if convention == 'casa':
        uvw = -uvw

    vis = np.zeros((len(uvw), len(freqs)), dtype=np.complex128)
    c = 299792458.  # m/s

    Nl, Nm = dirty.shape
    l = (l0 + (-Nl / 2 + np.arange(Nl)) * dl).reshape((Nl,1))
    m = (m0 + (-Nm / 2 + np.arange(Nm)) * dm).reshape((1,Nm))
    n = np.sqrt(1. - l**2 - m**2)
    nm1 = n-1

    for row, (u, v, w) in enumerate(uvw):
        for col, freq in enumerate(freqs):
            wavelength = c / freq
            phase = -2j * np.pi * (u * l + v * m + w * nm1) / wavelength
            vis[row, col] = np.sum(dirty * np.exp(phase) / n)
    return vis

@pytest.mark.parametrize("center_offset", [0.0, 0.1, 0.2])
@pytest.mark.parametrize("convention", ['physical', 'casa'])
@pytest.mark.parametrize("dirty_ordering", ['detector', 'world'])
def test_consistency(center_offset: float, convention: str, dirty_ordering: str):
    np.random.seed(42)
    Nl, Nm = 500, 700
    num_ants = 10
    num_freqs = 1

    pixsize = 0.5 * np.pi / 180 / 3600.  # 0.5 arcsec ~ 4 pixels / beam, so we'll avoid aliasing
    l0 = center_offset
    m0 = center_offset*0.8  # to break symmetry
    dl = pixsize
    dm = pixsize*0.9  # to break symmetry
    dirty = np.random.normal(size=(Nl, Nm))

    def pixel_to_lmn(xi, yi):
        l = l0 + (-Nl / 2 + xi) * dl
        m = m0 + (-Nm / 2 + yi) * dm
        n = np.sqrt(1. - l ** 2 - m ** 2)
        return np.asarray([l, m, n])

    antenna_1, antenna_2 = np.asarray(list(itertools.combinations(range(num_ants), 2))).T
    antennas = 10e3 * np.random.normal(size=(num_ants, 3))
    antennas[:, 2] *= 0.001
    uvw = antennas[antenna_2] - antennas[antenna_1]

    freqs = np.linspace(700e6, 2000e6, num_freqs)

    vis = new_dirty2vis(
        uvw=uvw,
        freqs=freqs,
        dirty=dirty,
        wgt=None,
        pixsize_l=dl,
        pixsize_m=dm,
        center_l=l0,
        center_m=m0,
        epsilon=1e-8,
        do_wgridding=True,
        divide_by_n=True,
        nthreads=1,
        verbosity=0,
        convention=convention,
        dirty_ordering=dirty_ordering,
    )

    vis_explicit = explicit_degridder(uvw, freqs, dirty, l0, dl, m0, dm, convention, dirty_ordering)

    np.testing.assert_allclose(vis.real, vis_explicit.real, atol=1e-4)
    np.testing.assert_allclose(vis.imag, vis_explicit.imag, atol=1e-4)

[aroffringa]

Maybe at least make both interfaces available? The old interface is imho better self-described, I'd prefer it over one with the non-well known terms. While it is true the terms can be documented, the documentation would basically just state how they call the old interface, so it's basically an introduction of new terminology for making only a wrapper.

I would call "dirty_ordering" just "image_ordering". It's not by definition a dirty image that comes out (it can be residual, plus it should also hold for the model image that is used as input, which is not "dirty").

I would not use 'casa' as a term. If you want a string-like configuration option, I would just limit it to 'physical', 'all_flipped', and 'negative_ra'. Those are the only three actual use-cases, isn't it?

[mreineck]

As I said, the original interface will not go away :-) The only change that may be necessary when calling fom C++ is the addition of flip_u and flip_w, which just adds to false arguments to the wgridder calls. (Yes, I could add the new flags at the end of the argument list and give them a default of false, but then the ordering of arguments wil become an even greater mess than it already is ...)

[landmanbester]

This is great, thanks @mreineck. I had a look and it seems that flip_[uvw] defaults to False for all three. Does this mean the the new dirty2vis in the experimental module will produce different results compared to the old interface when run with the defaults? I also don't see any documentation for the convention that is assumed for image coordinates. From the test above it looks like this hasn't changed. Is that correct? Would you consider documenting this somewhere?

[mreineck]

The default for flip_v was already False before, so the behaviour should not change at all in comparison to the earlier version. If you notice any situation where it does, please let me know!

I'm happy to document the status quo as clearly as I can ... maybe the best would be if one of you writes it down in a way that's clear to everyone in the radio community, and then I put this into the docstring. The only unambiguous thing I can do is paste the equivalent DFT code.

[landmanbester]

I was thinking if flip_w which is True in the test above. My test suite should answer this question though. Will let you know if I see any discrepancies

[mreineck]

In the test above it's set to True (unless you specify casa) because @Joshuaalbert needs it that way for the interface he wants. The default in the original routine ducc0.wgridder.experimental.dirty2vis is False, since that is how the currently released version also behaves.

[Joshuaalbert]

LGTM

[Joshuaalbert]

I think we can close this now.