ratt-ru / dask-ms

Implementation of a dask/xarray dataset backed by a CASA MS
https://dask-ms.readthedocs.io
Other
19 stars 7 forks source link

Support taql where clause #25

Closed sjperkins closed 5 years ago

sjperkins commented 5 years ago

Support TAQL where clause

sjperkins commented 5 years ago

Test cases pass on a standard run, but if only the test_taql_where case is run, the dread test.ms should be locked when using UserLocking reveals itself on my laptop:

$ py.test -s -v xarrayms/tests/ -k test_taql_where
=========================================================================================== test session starts ============================================================================================
platform linux2 -- Python 2.7.16, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /home/sperkins/venv/xms/bin/python
cachedir: .pytest_cache
rootdir: /home/sperkins/work/ska/code/xarray-ms, inifile: setup.cfg
collected 38 items / 37 deselected / 1 selected                                                                                                                                                            

xarrayms/tests/test_ms.py::test_taql_where[index_cols0] FAILED

================================================================================================= FAILURES =================================================================================================
_______________________________________________________________________________________ test_taql_where[index_cols0] _______________________________________________________________________________________

ms = '/tmp/pytest-of-sperkins/pytest-121/msdir/test.ms', index_cols = ['TIME', 'ANTENNA1', 'ANTENNA2']

    @pytest.mark.parametrize('index_cols', [
        ["TIME", "ANTENNA1", "ANTENNA2"]])
    def test_taql_where(ms, index_cols):
        xds = list(xds_from_table(ms, taql_where="FIELD_ID >= 0 AND FIELD_ID < 2",
                                  columns=["FIELD_ID"],
                                  table_kwargs={'ack': False}))

        assert len(xds) == 1
        assert (xds[0].FIELD_ID.data.compute() == [0, 0, 0, 1, 1, 1, 1]).all()

        xds = list(xds_from_table(ms, taql_where="FIELD_ID >= 0 AND FIELD_ID < 2",
                                  group_cols=["DATA_DESC_ID", "SCAN_NUMBER"],
                                  columns=["FIELD_ID"],
>                                 table_kwargs={'ack': False}))

xarrayms/tests/test_ms.py:315: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
xarrayms/xarray_ms.py:821: in xds_from_table
    with pt.taql(query) as gq:
../../../../venv/xms/local/lib/python2.7/site-packages/casacore/tables/table.py:156: in taql
    tab = table(cmd, tabs, _oper=2)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <casacore.tables.table.table object at 0x7f604d92e3c0>
tablename = 'using style Python SELECT DATA_DESC_ID, SCAN_NUMBER, GROWID() as __tablerow__ FROM $1 GROUPBY DATA_DESC_ID, SCAN_NUMBER HAVING FIELD_ID >= 0 AND FIELD_ID < 2'
tabledesc = [<casacore.tables.table.table object at 0x7f604d92e310>], nrow = 0, readonly = True, lockoptions = 'default', ack = True, dminfo = {}, endian = 'aipsrc', memorytable = False
concatsubtables = [], _columnnames = [], _datatypes = [], _oper = 2, _delete = False

    def __init__(self, tablename, tabledesc=False, nrow=0, readonly=True,
                 lockoptions='default', ack=True, dminfo={}, endian='aipsrc',
                 memorytable=False, concatsubtables=[],
                 _columnnames=[], _datatypes=[],
                 _oper=0, _delete=False):
        """Open or create a table."""
        if _oper == 1:
            # This is the readascii constructor.
            tabname = _remove_prefix(tablename)
            Table.__init__(self, tabname, tabledesc, nrow, readonly,
                           lockoptions, ack, dminfo, endian, memorytable,
                           _columnnames, _datatypes)
        elif _oper == 2:
            # This is the query or calc constructor.
>           Table.__init__(self, tablename, tabledesc)
E           RuntimeError: Error in TaQL command: 'using style Python SELECT DATA_DESC_ID, SCAN_NUMBER, GROWID() as __tablerow__ FROM $1 GROUPBY DATA_DESC_ID, SCAN_NUMBER HAVING FIELD_ID >= 0 AND FIELD_ID < 2'
E             ColumnSet::doLock: table /tmp/pytest-of-sperkins/pytest-121/msdir/test.ms should be locked when using UserLocking

../../../../venv/xms/local/lib/python2.7/site-packages/casacore/tables/table.py:331: RuntimeError
================================================================================= 1 failed, 37 deselected in 0.30 seconds ==================================================================================