Opening MS/Tables with read-only permissions

[Kitchi]

• dask-ms version: 0.2.1
• Python version: 3.6
• Operating System: Ubuntu

Description

I'm trying to open a measurement set that lives in a directory for which I don't have write access. This is intentional, since the data is the "original" copy of the observation and we don't accidentally want to write into it. However when I try to open it via dask-ms, it errors out because it cannot set a lock on the MS.

Is there a way to do open up the MS as "readonly"? I'm able to open things just fine as long as I have write access into the MS.

What I Did

from daskms import xds_from_ms
xds = xds_from_ms("/idia/projects/mightee/1523464709_1284.hh_vv.ms/")

and it errors out with

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/anaconda3/lib/python3.6/site-packages/daskms/table_proxy.py in __call__(cls, *args, **kwargs)
    168             try:
--> 169                 return _table_cache[key]
    170             except KeyError:

/anaconda3/lib/python3.6/weakref.py in __getitem__(self, key)
    136             self._commit_removals()
--> 137         o = self.data[key]()
    138         if o is None:

KeyError: -3464827575097077922

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-13-791da067da3d> in <module>()
----> 1 xds = xds_from_ms("/idia/projects/mightee/1523464709_1284.hh_vv.ms/")

/anaconda3/lib/python3.6/site-packages/daskms/dask_ms.py in xds_from_ms(ms, columns, index_cols, group_cols, **kwargs)
    327                           index_cols=index_cols,
    328                           group_cols=group_cols,
--> 329                           **kwargs)
    330 
    331 

/anaconda3/lib/python3.6/site-packages/daskms/dask_ms.py in xds_from_table(table_name, columns, index_cols, group_cols, **kwargs)
    254     dask_datasets = DatasetFactory(table_name, columns,
    255                                    group_cols, index_cols,
--> 256                                    **kwargs).datasets()
    257 
    258     # Return dask datasets if xarray is not available

/anaconda3/lib/python3.6/site-packages/daskms/reads.py in datasets(self)
    413         else:
    414             order_taql = group_ordering_taql(table_proxy, self.group_cols,
--> 415                                              self.index_cols, self.taql_where)
    416             orders = group_row_ordering(order_taql, self.group_cols,
    417                                         self.index_cols, self.chunks)

/anaconda3/lib/python3.6/site-packages/daskms/ordering.py in group_ordering_taql(table_proxy, group_cols, index_cols, taql_where)
    186 
    187         return TableProxy(taql_factory, query, tables=[table_proxy],
--> 188                           __executor_key__=table_proxy.executor_key)
    189 
    190     raise RuntimeError("Invalid condition in group_ordering_taql")

/anaconda3/lib/python3.6/site-packages/daskms/table_proxy.py in __call__(cls, *args, **kwargs)
    169                 return _table_cache[key]
    170             except KeyError:
--> 171                 instance = type.__call__(cls, *args, **kwargs)
    172                 _table_cache[key] = instance
    173                 return instance

/anaconda3/lib/python3.6/site-packages/daskms/table_proxy.py in __init__(self, factory, *args, **kwargs)
    291         # Private, should be inaccessible
    292         self._write = False
--> 293         self._writeable = ex.impl.submit(_iswriteable, table).result()
    294 
    295         should_be_writeable = not kwargs.get('readonly', True)

/anaconda3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

/anaconda3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

/anaconda3/lib/python3.6/concurrent/futures/thread.py in run(self)
     54 
     55         try:
---> 56             result = self.fn(*self.args, **self.kwargs)
     57         except BaseException as exc:
     58             self.future.set_exception(exc)

/anaconda3/lib/python3.6/site-packages/daskms/table_proxy.py in _iswriteable(table_future)
    237 
    238 def _iswriteable(table_future):
--> 239     return table_future.result().iswritable()
    240 
    241 

/anaconda3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    423                 raise CancelledError()
    424             elif self._state == FINISHED:
--> 425                 return self.__get_result()
    426 
    427             self._condition.wait(timeout)

/anaconda3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

/anaconda3/lib/python3.6/concurrent/futures/thread.py in run(self)
     54 
     55         try:
---> 56             result = self.fn(*self.args, **self.kwargs)
     57         except BaseException as exc:
     58             self.future.set_exception(exc)

/anaconda3/lib/python3.6/site-packages/daskms/table_proxy.py in taql_factory(query, style, tables)
    188 
    189     for t in tables:
--> 190         t.lock()
    191 
    192     try:

/anaconda3/lib/python3.6/site-packages/casacore/tables/table.py in lock(self, write, nattempts)
    723 
    724         """
--> 725         self._lock(write, nattempts)
    726 
    727     def unlock(self):

RuntimeError: Error (Bad file descriptor) when acquiring lock on /idia/projects/mightee/1523464709_1284.hh_vv.ms/table.lock

[sjperkins]

Thanks for the bug report @Kitchi.

I reproduced this by creating an MS as root in /tmp/root-ms/WSRT.MS and opening as another user with:

xds_from_ms("/tmp/root-ms/WSRT.MS")

https://github.com/ska-sa/dask-ms/pull/74 fixes my reproducer for me.

Would you confirm that this also works for your case by trying https://github.com/ska-sa/dask-ms/pull/74 out with your code?

[Kitchi]

Thanks for the quick response @sjperkins! I had issues building from code so I just pip install it. I'll try getting it to install from github again and let you know. Looking at the changes in the code I'm fairly sure it should work though.

[sjperkins]

You should just be able to update the python files for the sake of testing.

[sjperkins]

@kitchi I'm fairly convinced that #74 closes this so I've merged it into master. Feel free to re-open the issue if that doesn't fix your problem.

[Kitchi]

@sjperkins I can confirm that this works, thanks a lot!

Out of curiosity - will this change prevent me from writing back to the same MS (assuming I have write access)? It would be awesome if this was a switch in xds_from_table similar to astropy.io.fits.open

[sjperkins]

@sjperkins I can confirm that this works, thanks a lot!

Out of curiosity - will this change prevent me from writing back to the same MS (assuming I have write access)? It would be awesome if this was a switch in xds_from_table similar to astropy.io.fits.open

No, because table writes are setup using the xds_to_table function. Under the hood, xds_from_ms creates a readonly table object, while xds_to_table creates a writeable table object. They both refer to the same MS tho.

[Kitchi]

Okay awesome, thanks!