3.1.1 wheel installation does not bind correctly with casacore-data

[bennahugo]

Hi.

Sorry for the delay. Here are the minimum steps to reproduce this on an ubuntu 18.04 system:

  1 FROM kernsuite/base:5
  2
  3 RUN apt-get update
  4 RUN apt-get install -y git python-virtualenv casacore-dev casacore-data python-pip libboost-all-dev wcslib-dev libcfitsio-dev
  5                                                                                                                          
  6 WORKDIR /                                                                                                                
  7 RUN virtualenv venv
  8 RUN . venv/bin/activate && python2.7 -m pip install -U python-casacore==3.1.1
  9 RUN . venv/bin/activate && python2.7 -c "from casacore.measures import measures; print measures().get_observatories()"

yields

 > $ docker build -m 15g -t cc3.1 .                                                                                                                                              
Sending build context to Docker daemon  32.86MB
Step 1/7 : FROM kernsuite/base:5
 ---> befa68388c76
Step 2/7 : RUN apt-get update
 ---> Using cache
 ---> 210e8056ce47
Step 3/7 : RUN apt-get install -y git python-virtualenv casacore-dev casacore-data python-pip libboost-all-dev wcslib-dev libcfitsio-dev
 ---> Using cache
 ---> 9ab335b31358
Step 4/7 : WORKDIR /
 ---> Using cache
 ---> e7bd234fd6b1
Step 5/7 : RUN virtualenv venv
 ---> Using cache
 ---> 080a8e03f956
Step 6/7 : RUN . venv/bin/activate && python2.7 -m pip install -U python-casacore==3.1.1
 ---> Using cache
 ---> 4e068c4f5766
Step 7/7 : RUN . venv/bin/activate && python2.7 -c "from casacore.measures import measures; print measures().get_observatories()"
 ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
 ---> Running in 04416e457c00
2019-06-24 11:35:34     WARN    MeasIERS::findTab (file /build/casacore-3.1.1/measures/Measures/MeasIERS.cc, line 387)  Requested data table Observatories cannot be found in the searched directories:
2019-06-24 11:35:34     WARN    MeasIERS::findTab (file /build/casacore-3.1.1/measures/Measures/MeasIERS.cc, line 387)+ /build/ephemerides/
2019-06-24 11:35:34     WARN    MeasIERS::findTab (file /build/casacore-3.1.1/measures/Measures/MeasIERS.cc, line 387)+ /build/geodetic/
2019-06-24 11:35:34     SEVERE  MeasTable::doInitObservatories() (file /build/casacore-3.1.1/measures/Measures/MeasTable.cc, line 2865) Cannot read table of Observatories
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/local/lib/python2.7/site-packages/casacore/measures/__init__.py", line 869, in get_observatories
    return self.obslist()
RuntimeError: 2019-06-24 11:35:34       SEVERE  MeasTable::doInitObservatories() (file /build/casacore-3.1.1/measures/Measures/MeasTable.cc, line 2865) Cannot read table of Observatories
The command '/bin/sh -c . venv/bin/activate && python2.7 -c "from casacore.measures import measures; print measures().get_observatories()"' returned a non-zero code: 1

I can confirm that forcing wheel-rebuilding fixes the problem (e.g when installing into /usr/local on the container). It may be better not to try bundle binary that depends on system libraries for wheels since there are 0 guarantees that the wheels are going to be installed on a ubuntu derivative.

[bennahugo]

@ianheywood this replicates the problem you noted for your installations please ensure to use https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-no-binary as workaround

[tammojan]

Another workaround / solution is to edit ~/.casarc to point to the right location for the measures data.

[ludwigschwardt]

My 2 💰...

I would say use the following data directories for the various wheels:

• manylinux: /usr/share/casacore/data
• macOS: /usr/local/share/casacore/data

That's going to satisfy most of our users.

The only 100% clean solution would be a python-casacore Debian package / Homebrew formula that is guaranteed to work with the other parts of casacore in those distributions.

[ludwigschwardt]

@tammojan, interesting! Could you point me to more info on the .casarc file?

[tammojan]

@tammojan, interesting! Could you point me to more info on the .casarc file?

Sure. I'm afraid the documentation is both too concise and way too verbose, but have a look at the docs. The key for setting a custom data directory is measures.directory: ~/opt/casacore/data/ (for example, where this directory should contain a subdirectory geodetic).

[bennahugo]

ping on this issue. This also affects 3.2 wheels on a fresh installation. I've also seen binding issues against non-existing boost libraries.

  File "/home/hugo/ddf3debug/DDFacet/DDFacet/Data/ClassJones.py", line 876, in GiveBeam
    beam_times = self.BeamMachine.getBeamSampleTimes(times, quiet=quiet)
  File "/home/hugo/ddf3debug/DDFacet/DDFacet/Data/ClassFITSBeam.py", line 195, in getBeamSampleTimes
    dm.posangle(self.field_centre,self.zenith).get_value("deg") for t0 in beam_times ]
  File "/home/hugo/ddf3debug/DDFacet/DDFacet/Data/ClassFITSBeam.py", line 195, in <listcomp>
    dm.posangle(self.field_centre,self.zenith).get_value("deg") for t0 in beam_times ]
  File "/home/hugo/ddf3debug/venv/lib/python3.5/site-packages/casacore/measures/__init__.py", line 945, in posangle
    return _measures.posangle(self, m0, m1)
RuntimeError: 2020-01-14 06:59:07       SEVERE  MeasTable::dUTC(Double) (file /build/casacore-3.2.0/measures/Measures/MeasTable.cc, line 4281)  Cannot read leap second table TAI_UTC

[bennahugo]

Perhaps the solution is to rsync the NRAO data directly upon installation and do periodic rsyncs when the installation is out of date?

[ludwigschwardt]

AstroPy syncs its measure data to ~/.astropy/cache IIRC. That's another option.

[tammojan]

The following will fix your installation, which is not broken, just not configured to find data:

echo 'measures.directory: /usr/share/casacore/data' > ~/.casarc

It may be good to compile casacore in KERN the binary wheel build to specify this data directory, i.e. build casacore with -D DATA_DIR=/usr/share/casacore/data.