search

conda-forge / casacore-feedstock

A conda-smithy repository for casacore.
BSD 3-Clause "New" or "Revised" License
0 stars 11 forks source link

lib/casa/data not being collected #58

Open AlecThomson opened 3 years ago

AlecThomson commented 3 years ago

Issue: Hi there,

I hope this is the right place to raise this (apologies, if not).

The lib/casa directory isn't being created on installation. 

(spice) athomson@galaxy-2:/group/askap/athomson/miniconda3/envs/spice/lib> ls | grep casa
libcasa_casa.so
libcasa_casa.so.6
libcasa_coordinates.so
libcasa_coordinates.so.6
libcasa_derivedmscal.so
libcasa_derivedmscal.so.6
libcasa_fits.so
libcasa_fits.so.6
libcasa_images.so
libcasa_images.so.6
libcasa_lattices.so
libcasa_lattices.so.6
libcasa_meas.so
libcasa_meas.so.6
libcasa_measures.so
libcasa_measures.so.6
libcasa_mirlib.so
libcasa_mirlib.so.6
libcasa_msfits.so
libcasa_msfits.so.6
libcasa_ms.so
libcasa_ms.so.6
libcasa_python3.so
libcasa_python3.so.6
libcasa_scimath_f.so
libcasa_scimath_f.so.6
libcasa_scimath.so
libcasa_scimath.so.6
libcasa_tables.so
libcasa_tables.so.6

In particular, the lib/casa/data (and the data that lives in data) is missing which causes errors like:

2021-08-06 07:03:11     WARN    MeasIERS::findTab (file /home/conda/feedstock_root/build_artifacts/casacore_1613779202189/work/measures/Measures/MeasIERS.cc, line 387)  Requested data table TAI_UTC cannot be found in the searched directories:
2021-08-06 07:03:11     WARN    MeasIERS::findTab (file /home/conda/feedstock_root/build_artifacts/casacore_1613779202189/work/measures/Measures/MeasIERS.cc, line 387)+ /group/askap/athomson/miniconda3/envs/spice/lib/casa/data/ephemerides/
2021-08-06 07:03:11     WARN    MeasIERS::findTab (file /home/conda/feedstock_root/build_artifacts/casacore_1613779202189/work/measures/Measures/MeasIERS.cc, line 387)+ /group/askap/athomson/miniconda3/envs/spice/lib/casa/data/geodetic/
2021-08-06 07:03:11     SEVERE  MeasTable::dUTC(Double) (file /home/conda/feedstock_root/build_artifacts/casacore_1613779202189/work/measures/Measures/MeasTable.cc, line 4282)   Cannot read leap second table TAI_UTC

The compiler flag in build.sh seems to set the flag -DDATA_DIR=$PREFIX/lib/casa/data, so I'm not sure why it isn't being fetched.

Thanks!


Environment (conda list):

``` $ conda list # packages in environment at /group/askap/athomson/miniconda3/envs/spice: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge abseil-cpp 20210324.1 h9c3ff4c_0 conda-forge alsa-lib 1.2.3 h516909a_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argparse 1.4.0 pypi_0 pypi asciitree 0.3.3 py_2 asn1crypto 1.4.0 pyh9f0ad1d_0 conda-forge astroid 2.5 py38h06a4308_1 astropy 4.2 py38h497a2fe_2 astropy atk-1.0 2.36.0 h3371d22_4 conda-forge autopep8 1.5.7 pypi_0 pypi backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge bokeh 2.3.2 py38h578d9bd_0 conda-forge boost 1.74.0 py38hc10631b_3 conda-forge boost-cpp 1.74.0 hc6e9bd1_3 conda-forge brotlipy 0.7.0 py38h497a2fe_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.17.1 h7f98852_1 conda-forge ca-certificates 2021.5.30 ha878542_0 conda-forge cairo 1.16.0 h6cf1ce9_1008 conda-forge casacore 3.4.0 py38he0fa852_0 conda-forge certifi 2021.5.30 py38h578d9bd_0 conda-forge cffi 1.14.5 py38ha65f79e_0 conda-forge cfitsio 3.470 hb418390_7 conda-forge chardet 4.0.0 py38h578d9bd_1 conda-forge click 8.0.0 py38h578d9bd_0 conda-forge cloudpickle 1.6.0 py_0 conda-forge configargparse 1.5.1 pypi_0 pypi croniter 0.3.36 pyhd8ed1ab_0 conda-forge cryptography 3.4.7 py38ha5dfef3_0 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.11.0 py38h497a2fe_3 conda-forge dafits 0.0.2 pypi_0 pypi dask 2021.5.0 pyhd8ed1ab_0 conda-forge dask-core 2021.5.0 pyhd8ed1ab_0 conda-forge dask-ctl 2021.3.0 pyhd8ed1ab_0 conda-forge dask-jobqueue 0.7.2 pyhd8ed1ab_1 conda-forge dask-mpi 2.21.0 py38h4ecba47_1 conda-forge dbus 1.13.6 h48d8840_2 conda-forge decorator 5.0.9 pyhd8ed1ab_0 conda-forge distributed 2021.5.0 py38h578d9bd_0 conda-forge docker-py 5.0.0 py38h578d9bd_0 conda-forge docker-pycreds 0.4.0 py_0 conda-forge ephem 4.0.0.2 pypi_0 pypi expat 2.3.0 h9c3ff4c_0 conda-forge fasteners 0.16.3 pyhd3eb1b0_0 fftw 3.3.9 nompi_h74d3f13_101 conda-forge fitsio 1.1.4 py38h19b3c2e_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.13.1 hba837de_1005 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge freetype 2.10.4 h0708190_1 conda-forge fribidi 1.0.10 h36c2ea0_0 conda-forge frion 1.0 pypi_0 pypi fsspec 2021.5.0 pyhd8ed1ab_0 conda-forge future 0.18.2 py38h578d9bd_3 conda-forge gdk-pixbuf 2.42.6 h04a7f16_0 conda-forge gettext 0.19.8.1 h0b5b191_1005 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.68.2 h9c3ff4c_0 conda-forge glib-tools 2.68.2 h9c3ff4c_0 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge graphviz 2.47.1 h85b4f2f_1 conda-forge gst-plugins-base 1.18.4 hf529b03_2 conda-forge gstreamer 1.18.4 h76c114f_2 conda-forge gtk2 2.24.33 h539f30e_1 conda-forge gts 0.7.6 h64030ff_2 conda-forge harfbuzz 2.8.1 h83ec7ef_0 conda-forge hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge heapdict 1.0.1 py_0 conda-forge icu 68.1 h58526e2_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib_resources 5.1.3 py38h578d9bd_0 conda-forge ipython 7.23.1 py38hd0cf306_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge isort 5.8.0 pyhd3eb1b0_0 jedi 0.18.0 py38h578d9bd_2 conda-forge jinja2 3.0.1 pyhd8ed1ab_0 conda-forge joblib 1.0.1 pypi_0 pypi jpeg 9d h36c2ea0_0 conda-forge kiwisolver 1.3.1 py38h1fd1430_1 conda-forge krb5 1.19.1 hcc1bbae_0 conda-forge lazy-object-proxy 1.6.0 py38h27cfd23_0 lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge libblas 3.9.0 9_openblas conda-forge libcblas 3.9.0 9_openblas conda-forge libclang 11.1.0 default_ha53f305_1 conda-forge libcurl 7.76.1 h2574ce0_2 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 hcdb4288_3 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 9.3.0 h2828fa1_19 conda-forge libgd 2.3.2 h78a0170_0 conda-forge libgfortran-ng 9.3.0 hff62375_19 conda-forge libgfortran5 9.3.0 hff62375_19 conda-forge libglib 2.68.2 h3e27bee_0 conda-forge libgomp 9.3.0 h2828fa1_19 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.9.0 9_openblas conda-forge libllvm11 11.1.0 hf817b99_2 conda-forge libnghttp2 1.43.0 h812cca2_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopenblas 0.3.15 pthreads_h8fe5266_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.3 hd57d9b9_0 conda-forge librsvg 2.50.5 hc3c00ef_0 conda-forge libssh2 1.9.0 ha56f1ee_6 conda-forge libstdcxx-ng 9.3.0 h6de172a_19 conda-forge libtiff 4.2.0 hbd63e13_2 conda-forge libtool 2.4.6 h58526e2_1007 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp 1.2.0 h3452ae3_0 conda-forge libwebp-base 1.2.0 h7f98852_2 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxkbcommon 1.0.3 he3ba5ed_0 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.3 h9c3ff4c_0 conda-forge markupsafe 2.0.1 py38h497a2fe_0 conda-forge marshmallow 3.12.1 pyhd8ed1ab_0 conda-forge marshmallow-oneofschema 2.1.0 pyhd8ed1ab_0 conda-forge matplotlib 3.4.2 py38h578d9bd_0 conda-forge matplotlib-base 3.4.2 py38hcc49a3a_0 conda-forge matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge mccabe 0.6.1 py38_1 mongodb 4.2.14 h16cc2d0_1 conda-forge mpi 1.0 mpich conda-forge mpi4py 3.0.3 py38he865349_7 conda-forge mpich 3.4.1 h846660c_104 conda-forge msgpack-python 1.0.2 py38h1fd1430_1 conda-forge multinest 3.10 h8c654ad_8 conda-forge mypy 0.812 pypi_0 pypi mypy_extensions 0.4.3 py38h578d9bd_3 conda-forge mysql-common 8.0.23 ha770c72_2 conda-forge mysql-libs 8.0.23 h935591d_2 conda-forge natsort 7.1.1 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nspr 4.30 h9c3ff4c_0 conda-forge nss 3.65 hb5efdd6_0 conda-forge numcodecs 0.8.0 py38h2531618_0 numpy 1.20.2 py38h9894fe3_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1k h7f98852_0 conda-forge packaging 20.9 pyh44b312d_0 conda-forge pandas 1.2.4 py38h1abd341_0 conda-forge pango 1.48.5 hb8ff022_0 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pendulum 2.1.2 pyhd8ed1ab_1 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pgplot 5.2.2 h68245ad_1008 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.2.0 py38ha0e1e83_1 conda-forge pip 21.1.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge prefect 0.14.19 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.18 pyha770c72_0 conda-forge psutil 5.8.0 py38h497a2fe_1 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pycodestyle 2.7.0 pypi_0 pypi pycparser 2.20 pyh9f0ad1d_2 conda-forge pyephem 9.99 pypi_0 pypi pyerfa 1.7.2 py38h497a2fe_0 astropy pygments 2.9.0 pyhd8ed1ab_0 conda-forge pylint 2.7.4 py38h06a4308_1 pymongo 3.11.0 py38h709712a_4 conda-forge pymultinest 2.11 py38h578d9bd_0 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyqt 5.12.3 py38h578d9bd_7 conda-forge pyqt-impl 5.12.3 py38h7400c14_7 conda-forge pyqt5-sip 4.19.18 py38h709712a_7 conda-forge pyqtchart 5.12 py38h7400c14_7 conda-forge pyqtwebengine 5.12.1 py38h7400c14_7 conda-forge pysocks 1.7.1 py38h578d9bd_3 conda-forge python 3.8.10 h49503c6_1_cpython conda-forge python-box 5.3.0 pyhd8ed1ab_0 conda-forge python-casacore 3.4.0 py38h866ba35_0 conda-forge python-dateutil 2.8.1 py_0 conda-forge python-graphviz 0.16 pyh243d235_2 conda-forge python-slugify 5.0.2 pyhd8ed1ab_0 conda-forge python_abi 3.8 1_cp38 astropy pytz 2021.1 pyhd8ed1ab_0 conda-forge pytzdata 2020.1 pyh9f0ad1d_0 conda-forge pyyaml 5.4.1 py38h497a2fe_0 conda-forge qt 5.12.9 hda022c4_4 conda-forge radio-beam 0.3.3 pypi_0 pypi readline 8.1 h46c0cb4_0 conda-forge requests 2.25.1 pyhd3deb0d_0 conda-forge rm-tools 1.1.1 dev_0 rmextract 0.4.2 pypi_0 pypi ruamel.yaml 0.17.4 py38h497a2fe_0 conda-forge ruamel.yaml.clib 0.2.2 py38h497a2fe_2 conda-forge scipy 1.6.3 py38h7b17777_0 conda-forge setuptools 49.6.0 py38h578d9bd_3 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge spectral-cube 0.5.0 pypi_0 pypi spiceracs 0.1.0 dev_0 sqlite 3.35.5 h74cdb3f_0 conda-forge tabulate 0.8.9 pyhd8ed1ab_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge text-unidecode 1.3 py_0 conda-forge tk 8.6.10 h21135ba_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge toolz 0.11.1 py_0 conda-forge tornado 6.1 py38h497a2fe_1 conda-forge tqdm 4.60.0 pypi_0 pypi traitlets 5.0.5 py_0 conda-forge typed-ast 1.4.3 pypi_0 pypi typing_extensions 3.7.4.3 py_0 conda-forge unidecode 1.2.0 pyhd8ed1ab_0 conda-forge urllib3 1.26.4 pyhd8ed1ab_0 conda-forge wcslib 7.3 h72513aa_1 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge websocket-client 0.57.0 py38h578d9bd_4 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge wrapt 1.12.1 py38h7b6447c_1 xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.1 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge yaml-cpp 0.6.3 he1b5a44_4 conda-forge zarr 2.8.1 pyhd3eb1b0_0 zict 2.0.0 py_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.9 ha95c52a_0 conda-forge ```


Details about conda and system ( conda info ):

``` $ conda info active environment : spice active env location : /group/askap/athomson/miniconda3/envs/spice shell level : 2 user config file : /home/athomson/.condarc populated config files : /home/athomson/.condarc conda version : 4.10.3 conda-build version : not installed python version : 3.7.6.final.0 virtual packages : __linux=4.4.180=0 __glibc=2.22=0 __unix=0=0 __archspec=1=x86_64 base environment : /group/askap/athomson/miniconda3 (writable) conda av data dir : /group/askap/athomson/miniconda3/etc/conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /group/askap/athomson/miniconda3/pkgs /home/athomson/.conda/pkgs envs directories : /group/askap/athomson/miniconda3/envs /home/athomson/.conda/envs platform : linux-64 user-agent : conda/4.10.3 requests/2.22.0 CPython/3.7.6 Linux/4.4.180-94.141-default sles/12.3 glibc/2.22 UID:GID : 24456:24456 netrc file : None offline mode : False ```
pkgw commented 3 years ago

Yes, you're noticing a real problem. The CASA support data files are pretty hefty (about 300 MB from what I'm seeing) and they're not OS-dependent, so conda-forge ought to distribute them as a separate noarch package. But we don't have any package for them at all. (There are some technical reasons why it's not straightforward to package the data files directly in the casacore package, too.)

I have a personal package of the CASA data files. I haven't updated it for a while, but it should get you going:

https://anaconda.org/pkgw-forge/casa-data/

It would be good to integrate this package into conda-forge for real, although I don't know if there are any limits on package file sizes that might cause problems. My recipe is here:

https://github.com/pkgw/conda-recipes/tree/master/recipes/casa-data

If we integrate the data into conda-forge there would be a question as to whether to make casacore require casa-data. You can do a lot with the libraries without needing to tack on the 300 MB download, but if the user were to get one of the errors above, it'd be hard to convey to them how to solve the issue.