search

CDAT / cdms

8 stars 10 forks source link

setup.py creating multiple packages #410

Closed xylar closed 3 years ago

xylar commented 3 years ago

Describe the bug

Currently, setup.py is creating cdms2, MV2 and regrid2 with multiple calls to setup(). This is seemingly not considered a good practice: https://stackoverflow.com/questions/46118102/is-it-safe-to-call-setup-multiple-times-in-a-single-setup-py https://stackoverflow.com/questions/51296179/how-do-i-build-multiple-wheel-files-from-a-single-setup-py It interferes with the ability to install with pip install . and it does not allow each package to be installed individually (e.g in the conda-forge recipe).

The current approach means conda think that the regrid2, mv2 and cdms2 packages come from pypi, not conda-forge. It also means there is no way to specify dependencies for regrid2 and mv2 or to use conda search --info to find out anything about these packages.

My suggestion, following https://stackoverflow.com/questions/46118102/is-it-safe-to-call-setup-multiple-times-in-a-single-setup-py, would be to have multiple setup.py files, in different directories, one for each of cdms2 and regrid2. It isn't clear to me if it is worthwhile to keep mv2 as a separate package. It would seem better to just update any dependencies that currently expect it (including cdms2 itself) to use cdms.MV2 instead.

I would be happy to provide a PR for these changes if that is of interest.

To Reproduce Steps to reproduce the behavior:

  1. Run

    $ conda create -y -n test python=3.8 cdms2
$ conda activate test
$ conda list cdms2
$ conda list regrid2

    Note: mv2 is not listed as a separate package at all.

  2. See errors:

    
# packages in environment at /home/xylar/miniconda3/envs/test:
#
# Name                    Version                   Build  Channel
cdms2                     3.1.5                    pypi_0    pypi

packages in environment at /home/xylar/miniconda3/envs/test:

#

Name Version Build Channel

regrid2 3.1.5 pypi_0 pypi

3. Try to search for regrid2 on `conda-forge`:

$ conda search --info -c conda-forge regrid2

4. See the error:

Loading channels: done No match found for: regrid2. Search: regrid2

PackagesNotFoundError: The following packages are not available from current channels:

Current channels:

To search for alternate channels that may provide the conda package you're looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

**Expected behavior**

I would expect to see `cdms2`, `mv2` and `regird2` on the `conda-forge` channel.  I would expect to be able to search for and independently install `regrid2` and `mv2`.

**Desktop (please complete the following information):**
 - OS: Ubuntu 18.04

## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

$ conda info

 active environment : test
active env location : /home/xylar/miniconda3/envs/test
        shell level : 2
   user config file : /home/xylar/.condarc

populated config files : /home/xylar/.condarc conda version : 4.8.3 conda-build version : 3.19.2 python version : 3.7.6.final.0 virtual packages : __glibc=2.27 base environment : /home/xylar/miniconda3 (writable) channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /home/xylar/miniconda3/pkgs /home/xylar/.conda/pkgs envs directories : /home/xylar/miniconda3/envs /home/xylar/.conda/envs platform : linux-64 user-agent : conda/4.8.3 requests/2.24.0 CPython/3.7.6 Linux/4.15.0-1093-oem ubuntu/18.04.5 glibc/2.27 UID:GID : 1001:1001 netrc file : None offline mode : False

</p></details>

<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

$ conda config --show-sources ==> /home/xylar/.condarc <== channel_priority: strict channels:

`conda list --show-channel-urls`

``` $ conda list --show-channel-urls # packages in environment at /home/xylar/miniconda3/envs/test: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge attrs 19.3.0 py_0 conda-forge brotlipy 0.7.0 py38h1e0a361_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.16.1 h516909a_0 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge cdat_info 8.2.1 pyh9f0ad1d_1 conda-forge cdms2 3.1.5 pypi_0 pypi cdtime 3.1.4 py38hac60b08_0 conda-forge certifi 2020.6.20 py38h32f6830_0 conda-forge cffi 1.14.1 py38h5bae8af_0 conda-forge chardet 3.0.4 py38h32f6830_1006 conda-forge cryptography 3.0 py38h766eaa4_0 conda-forge curl 7.71.1 he644dc0_4 conda-forge decorator 4.4.2 py_0 conda-forge distarray 2.12.2 py_1 conda-forge esmf 8.0.1 nompi_hbeb3ca6_0 conda-forge esmpy 8.0.1 nompi_py38hbd9704b_0 conda-forge future 0.18.2 py38h32f6830_1 conda-forge g2clib 1.6.0 hf3f1b0b_9 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.6 nompi_h3c11f04_101 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 1.7.0 py38h32f6830_0 conda-forge importlib_metadata 1.7.0 0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jasper 1.900.1 h07fcdf6_1006 conda-forge jpeg 9d h516909a_0 conda-forge jsonschema 3.2.0 py38h32f6830_1 conda-forge jupyter_core 4.6.3 py38h32f6830_1 conda-forge krb5 1.17.1 hfafb76e_2 conda-forge lazy-object-proxy 1.5.1 py38h1e0a361_0 conda-forge ld_impl_linux-64 2.34 hc38a660_9 conda-forge libblas 3.8.0 17_openblas conda-forge libcblas 3.8.0 17_openblas conda-forge libcdms 3.1.2 h054cd8a_112 conda-forge libcf 1.0.3 py38h2f41aa0_108 conda-forge libcurl 7.71.1 hcdd3856_4 conda-forge libdrs 3.1.2 hc2e2db3_112 conda-forge libdrs_f 3.1.2 hae7e664_110 conda-forge libedit 3.1.20191231 h46ee950_1 conda-forge libev 4.33 h516909a_0 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.3.0 h24d8f2e_14 conda-forge libgfortran-ng 7.5.0 hdf63c60_14 conda-forge libgomp 9.3.0 h24d8f2e_14 conda-forge liblapack 3.8.0 17_openblas conda-forge libnetcdf 4.7.4 nompi_h84807e1_105 conda-forge libnghttp2 1.41.0 hab1572f_1 conda-forge libopenblas 0.3.10 pthreads_hb3c22a3_4 conda-forge libpng 1.6.37 hed695b0_1 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.3.0 hdf63c60_14 conda-forge libtiff 4.1.0 hc7e4089_6 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libwebp-base 1.1.0 h516909a_3 conda-forge lz4-c 1.9.2 he1b5a44_1 conda-forge nbformat 5.0.7 py_0 conda-forge ncurses 6.2 he1b5a44_1 conda-forge netcdf-fortran 4.5.3 nompi_hfef6a68_100 conda-forge numpy 1.19.1 py38h8854b6b_0 conda-forge openblas 0.3.10 pthreads_hf183345_4 conda-forge openssl 1.1.1g h516909a_1 conda-forge pip 20.2.1 py_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pyopenssl 19.1.0 py_1 conda-forge pyrsistent 0.16.0 py38h1e0a361_0 conda-forge pysocks 1.7.1 py38h32f6830_1 conda-forge python 3.8.5 h4d41432_2_cpython conda-forge python_abi 3.8 1_cp38 conda-forge readline 8.0 he28a2e2_2 conda-forge regrid2 3.1.5 pypi_0 pypi requests 2.24.0 pyh9f0ad1d_0 conda-forge setuptools 49.2.1 py38h32f6830_0 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sqlite 3.32.3 hcee41ef_1 conda-forge tk 8.6.10 hed695b0_0 conda-forge traitlets 4.3.3 py38h32f6830_1 conda-forge urllib3 1.25.10 py_0 conda-forge wheel 0.34.2 py_1 conda-forge xz 5.2.5 h516909a_1 conda-forge zipp 3.1.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.5 h6597ccf_2 conda-forge ```

xylar commented 3 years ago

I played around with splitting regrid2 and cdms2 into separate setup.py scripts and separate conda feedstocks. The problem is that each imports the other.

It seems to me that it would make sense to make regrid2 be part of cdms2, rather than a separate package. I know that would cascade to other CDAT packages, but it should be manageable. I would also stop having MV2 as a separate module/package. It would make things a lot cleaner on conda-forge.

Moving to using pip install instead of python setup.py install would likely also be helpful, but it is likely a follow-on issue.

@muryanto1 or others, I would be happy to chat more about this sometime when you're available.

jasonb5 commented 3 years ago

@xylar Thanks for opening this issue. This is definitely a long standing annoyance of the package. I agree, the path foward would be combining them, this is in-line with our next-gen CDMS design.

xylar commented 3 years ago

Okay, sounds great!