search

NCAR / ncar-conda

ncar-conda - YAML inventories for Conda environments on NCAR HPC systems
MIT License
1 stars 1 forks source link

Install `dask-mpi` and consider installing some packages to use MPI builds #4

Closed dphow closed 9 months ago

dphow commented 1 year ago

Installing dask-mpi has minimal impact on current NPL. See below:

Transaction

  Prefix: /glade/work/dhoward/conda-envs/test_dask

  Updating specs:

   - dask-mpi
   - ca-certificates
   - certifi
   - openssl

  Package      Version  Build         Channel           Size
──────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────

  + dask-mpi  2022.4.0  pyh458ca06_2  conda-forge     Cached

Also, consider pinning or installing mpi versions of common packages. Note the current packages that don't use mpi:

mamba list | grep mpi
dask-mpi                  2022.4.0           pyh458ca06_2    conda-forge
esmf                      8.4.2           nompi_ha7f9e30_1    conda-forge
h5py                      3.9.0           nompi_py310hcca72df_101    conda-forge
hdf5                      1.14.1          nompi_h4f84152_100    conda-forge
libnetcdf                 4.9.2           nompi_h78c856c_106    conda-forge
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.4           py310h37cc914_0    conda-forge
mpich                     4.1.1              h846660c_100    conda-forge
netcdf-fortran            4.6.1           nompi_hec59055_101    conda-forge
netcdf4                   1.6.4           nompi_py310h6f5dce6_101    conda-forge

And resulting changes if mamba install libnetcdf=*=mpi* h5py=*=mpi* netcdf4=*=mpi* is run:

Transaction

  Prefix: /glade/work/dhoward/conda-envs/test_dask

  Updating specs:

   - libnetcdf=*[build=mpi*]
   - h5py=*[build=mpi*]
   - netcdf4=*[build=mpi*]
   - ca-certificates
   - certifi
   - openssl

  Package           Version  Build                      Channel           Size
────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────

  + parallelio        2.6.0  mpi_mpich_h79a22d0_100     conda-forge      374kB

  Change:
────────────────────────────────────────────────────────────────────────────────

  - esmf              8.4.2  nompi_ha7f9e30_1           conda-forge     Cached
  + esmf              8.4.2  mpi_mpich_h8fb3fe9_101     conda-forge       24MB
  - h5py              3.9.0  nompi_py310hcca72df_101    conda-forge     Cached
  + h5py              3.9.0  mpi_mpich_py310hdfe7061_1  conda-forge        1MB
  - hdf5             1.14.1  nompi_h4f84152_100         conda-forge     Cached
  + hdf5             1.14.1  mpi_mpich_ha2c2bf8_0       conda-forge        4MB
  - libnetcdf         4.9.2  nompi_h78c856c_106         conda-forge     Cached
  + libnetcdf         4.9.2  mpi_mpich_hd696bd7_7       conda-forge      848kB
  - netcdf-fortran    4.6.1  nompi_hec59055_101         conda-forge     Cached
  + netcdf-fortran    4.6.1  mpi_mpich_h17322d4_1       conda-forge      433kB
  - netcdf4           1.6.4  nompi_py310h6f5dce6_101    conda-forge     Cached
  + netcdf4           1.6.4  mpi_mpich_py310h628a664_1  conda-forge      501kB
vanderwb commented 1 year ago

Thanks again Daniel. These improvements seem logical to me. I can't think of any side-effects from doing this, but I'll dig a bit more and make sure before a Monday update roll-up.

vanderwb commented 1 year ago

Resolved in two latest commits (21f19e8 and 548f61a).