Add micromamba support - Githubissues

Bai-Chiang commented 10 months ago

Let soconda.sh work with micromanba installation.

close previous PR https://github.com/simonsobs/soconda/pull/14. With new configurable setup, there is no need to separate installation steps.

Change log

Add micromamba support. Use conda_exec wraps conda or micromamba
Bump python version in minimal config following PR #13
Add missing pip packages (qpoint, posix-ipc) to minimal config.
Fix sotodlib link in minimal and perlmutter config
Update document. Since miniforge now use libmamba by default [1] no need for extra script.
Sanitize -e envname input. If it's a path, make sure it's full path. (Bobby tables)
Rearrange codes such that logically similar parts groups together
Use $scriptdir/tmpfs as TMPDIR instead of default /tmp. This make clean up easier. Also remove the tmp directory after building conda packages. Also remove /tmp/pixell-* files.
Simplify conda packages installation by reading packages.txt directly.
Replace cmd | tee -a "log" 2>&1 with cmd |& tee -a "log" (equivalent to cmd 2>&1 | tee -a "log"), the old one does not log stderr correctly.
Speed up pip packages installation, by running conda list only once, instead of running for each dependency. conda list command takes ~1s. Some packages like toast so3g pixell have 20+ dependencies this could save quite some time.
Add some echo -e "\n\n" to make stdout look cleaner.

[1] https://github.com/conda-forge/miniforge#whats-the-difference-between-mambaforge-and-miniforge

tskisner commented 10 months ago

Ok, you should be able to rebase against main now that #13 is merged.

Bai-Chiang commented 10 months ago

Currently got some error when running the run_tests.sh

I'm using python=3.10 and pyfftw=0.12

minimal configuration (all binary installation) passed the test.
default configuration, got error when MPI enabled.

Same error on both micromamba and conda installation. Tested on two machines still same error. (Good at least it's reproducible.)

Error message:

test_perturbhwp_stepped (toast.tests.ops_perturbhwp.PerturbHWPTest) ... [3]ok TOAST INFO: Adding patch "small_patch"
TOAST INFO: Rectangular format
TOAST INFO: Creating '/tmp/tmp2qoukk9_'
TOAST INFO: Loading schedule from /tmp/tmp2qoukk9_/ground_schedule.txt
TOAST INFO: Loaded 23 scans from /tmp/tmp2qoukk9_/ground_schedule.txt totaling 4.25 hours.
[3]error [2]error Proc 3: Traceback (most recent call last):
Proc 3:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
Proc 3:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
Proc 3:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
Proc 3:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/toast/tests/ops_perturbhwp.py", line 90, in test_perturbhwp_stepped
    orig[:] = np.round(np.linspace(0, 10, times.size)) * np.pi / 8
Proc 3:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/pshmem/shmem.py", line 291, in __setitem__
    raise RuntimeError(msg)
Proc 3: RuntimeError: When setting data with [] notation, there were 2 processes with a non-None value for the data instead of one process.

Proc 2: Traceback (most recent call last):
Proc 2:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
Proc 2:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
Proc 2:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
Proc 2:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/toast/tests/ops_perturbhwp.py", line 90, in test_perturbhwp_stepped
    orig[:] = np.round(np.linspace(0, 10, times.size)) * np.pi / 8
Proc 2:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/pshmem/shmem.py", line 291, in __setitem__
    raise RuntimeError(msg)
Proc 2: RuntimeError: When setting data with [] notation, there were 2 processes with a non-None value for the data instead of one process.

Proc 1: Traceback (most recent call last):
Proc 1:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
Proc 1:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
Proc 1:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
Proc 1:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/toast/tests/ops_perturbhwp.py", line 90, in test_perturbhwp_stepped
    orig[:] = np.round(np.linspace(0, 10, times.size)) * np.pi / 8
Proc 1:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/pshmem/shmem.py", line 291, in __setitem__
    raise RuntimeError(msg)
Proc 1: RuntimeError: When setting data with [] notation, there were 2 processes with a non-None value for the data instead of one process.

[1]error Proc 0: Traceback (most recent call last):
Proc 0:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 59, in testPartExecutor
    yield
Proc 0:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 591, in run
    self._callTestMethod(testMethod)
Proc 0:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/unittest/case.py", line 549, in _callTestMethod
    method()
Proc 0:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/toast/tests/ops_perturbhwp.py", line 90, in test_perturbhwp_stepped
    orig[:] = np.round(np.linspace(0, 10, times.size)) * np.pi / 8
Proc 0:   File "/home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/lib/python3.10/site-packages/pshmem/shmem.py", line 291, in __setitem__
    raise RuntimeError(msg)
Proc 0: RuntimeError: When setting data with [] notation, there were 2 processes with a non-None value for the data instead of one process.

[0]error --------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

environment.yaml

name: soconda-default_0.0.5.dev141
channels:
  - file:///home/tux/miniforge3/envs/soconda-default_0.0.5.dev141/conda-bld
  - conda-forge
  - nodefaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_kmp_llvm
  - alsa-lib=1.2.10=hd590300_0
  - amqp=5.2.0=pyhd8ed1ab_1
  - ansi2html=1.9.1=py310hff52083_0
  - anyio=4.2.0=pyhd8ed1ab_0
  - anytree=2.8.0=pyh9f0ad1d_0
  - aom=3.7.1=h59595ed_0
  - archspec=0.2.2=pyhd8ed1ab_0
  - argon2-cffi=23.1.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py310h2372a71_4
  - arrow=1.3.0=pyhd8ed1ab_0
  - astropy=6.0.0=py310h1f7b6fc_0
  - astropy-iers-data=0.2024.1.8.0.30.55=pyhd8ed1ab_0
  - asttokens=2.4.1=pyhd8ed1ab_0
  - async-lru=2.0.4=pyhd8ed1ab_0
  - attr=2.5.1=h166bdaf_1
  - attrs=23.2.0=pyh71513ae_0
  - autoconf=2.71=pl5321h2b4cb7a_1
  - automake=1.16.5=pl5321ha770c72_0
  - babel=2.14.0=pyhd8ed1ab_0
  - backports.zoneinfo=0.2.1=py310hff52083_8
  - beautifulsoup4=4.12.2=pyha770c72_0
  - billiard=4.2.0=py310h2372a71_0
  - binutils=2.40=hdd6e379_0
  - binutils_impl_linux-64=2.40=hf600244_0
  - binutils_linux-64=2.40=hbdbef99_2
  - bleach=6.1.0=pyhd8ed1ab_0
  - blinker=1.7.0=pyhd8ed1ab_0
  - blosc=1.21.5=h0f2a231_0
  - boltons=23.1.1=pyhd8ed1ab_0
  - brotli=1.1.0=hd590300_1
  - brotli-bin=1.1.0=hd590300_1
  - brotli-python=1.1.0=py310hc6cd4ac_1
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=hd590300_5
  - c-ares=1.25.0=hd590300_0
  - c-blosc2=2.12.0=hb4ffafa_0
  - c-compiler=1.7.0=hd590300_0
  - ca-certificates=2023.11.17=hbcca054_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - cairo=1.18.0=h3faef2a_0
  - celery=5.3.5=pyhd8ed1ab_0
  - certifi=2023.11.17=pyhd8ed1ab_0
  - cffi=1.16.0=py310h2fee648_0
  - cfitsio=4.3.1=hbdc6101_0
  - chardet=5.2.0=py310hff52083_1
  - charls=2.4.2=h59595ed_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - click-didyoumean=0.3.0=pyhd8ed1ab_0
  - click-plugins=1.1.1=py_0
  - click-repl=0.3.0=pyhd8ed1ab_0
  - cmake=3.28.1=hcfe8598_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - comm=0.2.1=pyhd8ed1ab_0
  - compilers=1.7.0=ha770c72_0
  - conda=23.11.0=py310hff52083_1
  - conda-build=3.28.3=py310hff52083_0
  - conda-index=0.3.0=pyhd8ed1ab_1
  - conda-libmamba-solver=23.12.0=pyhd8ed1ab_0
  - conda-package-handling=2.2.0=pyh38be061_0
  - conda-package-streaming=0.9.0=pyhd8ed1ab_0
  - contourpy=1.2.0=py310hd41b1e2_0
  - coverage=7.4.0=py310h2372a71_0
  - coveralls=1.8.0=py_0
  - cxx-compiler=1.7.0=h00ab1b0_0
  - cycler=0.12.1=pyhd8ed1ab_0
  - cython=3.0.7=py310hc6cd4ac_0
  - dash=2.14.2=pyhd8ed1ab_0
  - dash-core-components=2.0.0=pyhd8ed1ab_1
  - dash-html-components=2.0.0=pyhd8ed1ab_1
  - dash-table=5.0.0=pyhd8ed1ab_1
  - dav1d=1.2.1=hd590300_0
  - dbus=1.13.6=h5008d03_3
  - debugpy=1.8.0=py310hc6cd4ac_1
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - distro=1.9.0=pyhd8ed1ab_0
  - docopt=0.6.2=py_1
  - ducc0=0.33.0=py310hc6cd4ac_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - ephem=4.1.5=py310h2372a71_1
  - exceptiongroup=1.2.0=pyhd8ed1ab_2
  - executing=2.0.1=pyhd8ed1ab_0
  - expat=2.5.0=hcb278e6_1
  - fftw=3.3.10=nompi_hc118613_108
  - filelock=3.13.1=pyhd8ed1ab_0
  - flask=3.0.0=pyhd8ed1ab_0
  - flask-cors=4.0.0=pyhd8ed1ab_0
  - fmt=10.1.1=h00ab1b0_1
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=h77eed37_1
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - fonttools=4.47.2=py310h2372a71_0
  - fortran-compiler=1.7.0=heb67821_0
  - fqdn=1.5.1=pyhd8ed1ab_0
  - freetype=2.12.1=h267a509_2
  - future=0.18.3=pyhd8ed1ab_0
  - gcc=12.3.0=h8d2909c_2
  - gcc_impl_linux-64=12.3.0=he2b93b0_3
  - gcc_linux-64=12.3.0=h76fc315_2
  - gettext=0.21.1=h27087fc_0
  - gfortran=12.3.0=h499e0f7_2
  - gfortran_impl_linux-64=12.3.0=hfcedea8_3
  - gfortran_linux-64=12.3.0=h7fe76b4_2
  - giflib=5.2.1=h0b41bf4_3
  - glib=2.78.3=hfc55251_0
  - glib-tools=2.78.3=hfc55251_0
  - gmp=6.3.0=h59595ed_0
  - graphite2=1.3.13=he1b5a44_1001
  - greenlet=3.0.3=py310hc6cd4ac_0
  - gsl=2.7=he838d99_0
  - gst-plugins-base=1.22.8=h8e1006c_1
  - gstreamer=1.22.8=h98fc4e7_1
  - gxx=12.3.0=h8d2909c_2
  - gxx_impl_linux-64=12.3.0=he2b93b0_3
  - gxx_linux-64=12.3.0=h8a814eb_2
  - h5py=3.10.0=nompi_py310h65828d5_101
  - harfbuzz=8.3.0=h3d44ed6_0
  - hdf5=1.14.3=nompi_h4f84152_100
  - healpy=1.16.6=py310h510b526_2
  - icu=73.2=h59595ed_0
  - idna=3.6=pyhd8ed1ab_0
  - imagecodecs=2024.1.1=py310h496a806_0
  - imageio=2.33.1=pyh8c1a49c_0
  - importlib-metadata=7.0.1=pyha770c72_0
  - importlib_metadata=7.0.1=hd8ed1ab_0
  - importlib_resources=6.1.1=pyhd8ed1ab_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - ipykernel=6.28.0=pyhd33586a_0
  - ipython=8.20.0=pyh707e725_0
  - ipywidgets=8.1.1=pyhd8ed1ab_0
  - isoduration=20.11.0=pyhd8ed1ab_0
  - itsdangerous=2.1.2=pyhd8ed1ab_0
  - jedi=0.19.1=pyhd8ed1ab_0
  - jinja2=3.1.3=pyhd8ed1ab_0
  - jplephem=2.21=pyh864a33b_0
  - json5=0.9.14=pyhd8ed1ab_0
  - jsonpatch=1.33=pyhd8ed1ab_0
  - jsonpointer=2.4=py310hff52083_3
  - jsonschema=4.20.0=pyhd8ed1ab_0
  - jsonschema-specifications=2023.12.1=pyhd8ed1ab_0
  - jsonschema-with-format-nongpl=4.20.0=pyhd8ed1ab_0
  - jupyter-dash=0.4.2=pyhd8ed1ab_1
  - jupyter-lsp=2.2.1=pyhd8ed1ab_0
  - jupyter_client=8.6.0=pyhd8ed1ab_0
  - jupyter_core=5.7.1=py310hff52083_0
  - jupyter_events=0.9.0=pyhd8ed1ab_0
  - jupyter_server=2.12.4=pyhd8ed1ab_0
  - jupyter_server_terminals=0.5.1=pyhd8ed1ab_0
  - jupyterlab=4.0.10=pyhd8ed1ab_0
  - jupyterlab_pygments=0.3.0=pyhd8ed1ab_0
  - jupyterlab_server=2.25.2=pyhd8ed1ab_0
  - jupyterlab_widgets=3.0.9=pyhd8ed1ab_0
  - jxrlib=1.1=hd590300_3
  - kernel-headers_linux-64=2.6.32=he073ed8_16
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.5=py310hd41b1e2_1
  - kombu=5.3.3=py310hff52083_0
  - krb5=1.21.2=h659d440_0
  - lame=3.100=h166bdaf_1003
  - lazy_loader=0.3=pyhd8ed1ab_0
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libaatm=1.0.9=he1b5a44_0
  - libactpol=c0c1647ad62b418f34fe38eb73168d4f2e13ff6f=h9bf148f_0
  - libactpol_deps=db0aee380dad503ba8fdf058d4d8075387100758=h3218e01_0
  - libaec=1.1.2=h59595ed_1
  - libarchive=3.7.2=h039dbb9_0
  - libavif16=1.0.3=hef5bec9_1
  - libblas=3.9.0=20_linux64_openblas
  - libbrotlicommon=1.1.0=hd590300_1
  - libbrotlidec=1.1.0=hd590300_1
  - libbrotlienc=1.1.0=hd590300_1
  - libcap=2.69=h0f662aa_0
  - libcblas=3.9.0=20_linux64_openblas
  - libclang=15.0.7=default_hb11cfb5_4
  - libclang13=15.0.7=default_ha2b6cf4_4
  - libcups=2.3.3=h4637d8d_4
  - libcurl=8.5.0=hca28451_0
  - libdeflate=1.19=hd590300_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=hd590300_2
  - libevent=2.1.12=hf998b51_1
  - libexpat=2.5.0=hcb278e6_1
  - libffi=3.4.2=h7f98852_5
  - libflac=1.4.3=h59595ed_0
  - libgcc-devel_linux-64=12.3.0=h8bca6fd_103
  - libgcc-ng=13.2.0=h807b86a_3
  - libgcrypt=1.10.3=hd590300_0
  - libgfortran-ng=13.2.0=h69a702a_3
  - libgfortran5=13.2.0=ha4646dd_3
  - libglib=2.78.3=h783c2da_0
  - libgomp=13.2.0=h807b86a_3
  - libgpg-error=1.47=h71f35ed_0
  - libhwloc=2.9.3=default_h554bfaf_1009
  - libiconv=1.17=hd590300_2
  - libjpeg-turbo=3.0.0=hd590300_1
  - liblapack=3.9.0=20_linux64_openblas
  - liblapacke=3.9.0=20_linux64_openblas
  - liblief=0.12.3=h27087fc_0
  - libllvm14=14.0.6=hcd5def8_4
  - libllvm15=15.0.7=h5cf9203_3
  - libmamba=1.5.6=had39da4_0
  - libmambapy=1.5.6=py310h39ff949_0
  - libnghttp2=1.58.0=h47da74e_1
  - libnsl=2.0.1=hd590300_0
  - libogg=1.3.4=h7f98852_1
  - libopenblas=0.3.25=openmp_h2e9423c_0
  - libopus=1.3.1=h7f98852_1
  - libpng=1.6.39=h753d276_0
  - libpq=16.1=h33b98f1_7
  - libsanitizer=12.3.0=h0f45ef3_3
  - libsndfile=1.2.2=hc60ed4a_1
  - libsodium=1.0.18=h36c2ea0_1
  - libsolv=0.7.27=hfc55251_0
  - libsqlite=3.44.2=h2797004_0
  - libssh2=1.11.0=h0841786_0
  - libstdcxx-devel_linux-64=12.3.0=h8bca6fd_103
  - libstdcxx-ng=13.2.0=h7e041cc_3
  - libsystemd0=255=h3516f8a_0
  - libtiff=4.6.0=ha9c0a0a_2
  - libtool=2.4.7=h27087fc_0
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.46.0=hd590300_0
  - libvorbis=1.3.7=h9c3ff4c_0
  - libwebp-base=1.3.2=hd590300_0
  - libxcb=1.15=h0b41bf4_0
  - libxcrypt=4.4.36=hd590300_1
  - libxkbcommon=1.6.0=h5d7e998_0
  - libxml2=2.11.6=h232c23b_0
  - libzlib=1.2.13=hd590300_5
  - libzopfli=1.0.3=h9c3ff4c_0
  - llvm-openmp=17.0.6=h4dfa4b3_0
  - llvmlite=0.41.1=py310h1b8f574_0
  - lttbc=0.2.4=py310h1f7b6fc_1
  - lz4-c=1.9.4=hcb278e6_0
  - lzo=2.10=h516909a_1000
  - m4=1.4.18=h516909a_1001
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markupsafe=2.1.3=py310h2372a71_1
  - matplotlib=3.8.2=py310hff52083_0
  - matplotlib-base=3.8.2=py310h62c0568_0
  - matplotlib-inline=0.1.6=pyhd8ed1ab_0
  - mdurl=0.1.2=pyhd8ed1ab_0
  - menuinst=2.0.2=py310hff52083_0
  - metis=5.1.1=h59595ed_2
  - mistune=3.0.2=pyhd8ed1ab_0
  - moby2=f0056ac3618b8c4c96130fcb41f7be3d6bd07241=py310h9bf148f_0
  - more-itertools=10.2.0=pyhd8ed1ab_0
  - mpfr=4.2.1=h9458935_0
  - mpg123=1.32.4=h59595ed_0
  - mpi=1.0=openmpi
  - mpi4py=3.1.5=py310h2a790f2_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - mysql-common=8.0.33=hf1915f5_6
  - mysql-libs=8.0.33=hca2cd23_6
  - nbclient=0.8.0=pyhd8ed1ab_0
  - nbconvert-core=7.14.1=pyhd8ed1ab_0
  - nbformat=5.9.2=pyhd8ed1ab_0
  - ncurses=6.4=h59595ed_2
  - nest-asyncio=1.5.8=pyhd8ed1ab_0
  - networkx=3.2.1=pyhd8ed1ab_0
  - notebook-shim=0.2.3=pyhd8ed1ab_0
  - nspr=4.35=h27087fc_0
  - nss=3.96=h1d7d5a4_0
  - numba=0.58.1=py310h7dc5dd1_0
  - numpy=1.26.3=py310hb13e2d6_0
  - openblas=0.3.25=openmp_h82c68e4_0
  - openjpeg=2.5.0=h488ebb8_3
  - openmpi=4.1.6=hc5af2df_101
  - openssl=3.2.0=hd590300_1
  - orjson=3.9.10=py310h1e2579a_0
  - overrides=7.4.0=pyhd8ed1ab_0
  - packaging=23.2=pyhd8ed1ab_0
  - pandas=2.1.4=py310hcc13569_0
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - parso=0.8.3=pyhd8ed1ab_0
  - patch=2.7.6=h7f98852_1002
  - patchelf=0.17.2=h58526e2_0
  - pcre2=10.42=hcad00b1_0
  - perl=5.32.1=7_hd590300_perl5
  - pexpect=4.8.0=pyh1a96a4e_2
  - pgplot=5.2.2=hf4b3394_1008
  - pickleshare=0.7.5=py_1003
  - pillow=10.2.0=py310h01dd4db_0
  - pip=23.3.2=pyhd8ed1ab_0
  - pixell=0.21.1=py310h6578b86_0
  - pixman=0.43.0=h59595ed_0
  - pkginfo=1.9.6=pyhd8ed1ab_0
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
  - platformdirs=4.1.0=pyhd8ed1ab_0
  - plotly=5.18.0=pyhd8ed1ab_0
  - plotly-resampler=0.8.3.2=pyhd8ed1ab_0
  - pluggy=1.3.0=pyhd8ed1ab_0
  - ply=3.11=py_1
  - prometheus_client=0.19.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.42=pyha770c72_0
  - prompt_toolkit=3.0.42=hd8ed1ab_0
  - pshmem=0.2.14=pyhd8ed1ab_0
  - psutil=5.9.7=py310h2372a71_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pulseaudio-client=16.1=hb77b528_5
  - pure_eval=0.2.2=pyhd8ed1ab_0
  - py-lief=0.12.3=py310hd8f1fbe_0
  - pyaml=23.12.0=pyhd8ed1ab_0
  - pybind11-abi=4=hd8ed1ab_3
  - pycosat=0.6.6=py310h2372a71_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pyerfa=2.0.1.1=py310h1f7b6fc_0
  - pyfftw=0.12.0=py310h0885ff1_3
  - pygments=2.17.2=pyhd8ed1ab_0
  - pyparsing=3.1.1=pyhd8ed1ab_0
  - pyqt=5.15.9=py310h04931ad_5
  - pyqt5-sip=12.12.2=py310hc6cd4ac_5
  - pysocks=1.7.1=pyha2e5f31_6
  - pysqlite3=0.4.6=pyh911a504_0
  - pytest=7.4.4=pyhd8ed1ab_0
  - pytest-cov=4.1.0=pyhd8ed1ab_0
  - pytest-runner=6.0.0=pyhd8ed1ab_0
  - python=3.10.13=hd12c33a_1_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-fastjsonschema=2.19.1=pyhd8ed1ab_0
  - python-json-logger=2.0.7=pyhd8ed1ab_0
  - python-libarchive-c=5.0=py310hff52083_2
  - python-tzdata=2023.4=pyhd8ed1ab_0
  - python_abi=3.10=4_cp310
  - pytz=2023.3.post1=pyhd8ed1ab_0
  - pywavelets=1.4.1=py310h1f7b6fc_1
  - pyyaml=6.0.1=py310h2372a71_1
  - pyzmq=25.1.2=py310h795f18f_0
  - qpoint=1.12.4=py310h9bf148f_0
  - qt-main=5.15.8=h82b777d_17
  - rav1e=0.6.6=he8a937b_2
  - readline=8.2=h8228510_1
  - referencing=0.32.1=pyhd8ed1ab_0
  - reproc=14.2.4.post0=hd590300_1
  - reproc-cpp=14.2.4.post0=h59595ed_1
  - requests=2.31.0=pyhd8ed1ab_0
  - retrying=1.3.3=py_2
  - rfc3339-validator=0.1.4=pyhd8ed1ab_0
  - rfc3986-validator=0.1.1=pyh9f0ad1d_0
  - rhash=1.4.4=hd590300_0
  - rich=13.7.0=pyhd8ed1ab_0
  - ripgrep=14.0.3=he8a937b_1
  - rpds-py=0.16.2=py310hcb5633a_0
  - ruamel.yaml=0.18.5=py310h2372a71_0
  - ruamel.yaml.clib=0.2.7=py310h2372a71_2
  - scikit-image=0.22.0=py310hcc13569_2
  - scipy=1.11.4=py310hb13e2d6_0
  - send2trash=1.8.2=pyh41d4057_0
  - setuptools=69.0.3=pyhd8ed1ab_0
  - sgp4=2.22=py310hfb6f7a9_0
  - shellingham=1.5.4=pyhd8ed1ab_0
  - sip=6.7.12=py310hc6cd4ac_0
  - six=1.16.0=pyh6c4a22f_0
  - skyfield=1.45=pyh1a96a4e_0
  - snappy=1.1.10=h9fff704_0
  - sniffio=1.3.0=pyhd8ed1ab_0
  - soupsieve=2.5=pyhd8ed1ab_1
  - sqlalchemy=2.0.25=py310h2372a71_0
  - sqlite=3.44.2=h2c6b66d_0
  - stack_data=0.6.2=pyhd8ed1ab_0
  - suitesparse=5.10.1=h3ec001c_2
  - svt-av1=1.8.0=h59595ed_0
  - sysroot_linux-64=2.12=he073ed8_16
  - tbb=2021.11.0=h00ab1b0_0
  - tenacity=8.2.3=pyhd8ed1ab_0
  - terminado=0.18.0=pyh0d859eb_0
  - tifffile=2023.12.9=pyhd8ed1ab_0
  - tinycss2=1.2.1=pyhd8ed1ab_0
  - tk=8.6.13=noxft_h4845f30_101
  - toast=3980cbb579de56a379df000df76e69a1c1ac73ae=py310h3fd9d12_0
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - tomlkit=0.12.3=pyha770c72_0
  - tornado=6.3.3=py310h2372a71_1
  - tqdm=4.66.1=pyhd8ed1ab_0
  - trace-updater=0.0.9.1=pyhd8ed1ab_0
  - traitlets=5.14.1=pyhd8ed1ab_0
  - truststore=0.8.0=pyhd8ed1ab_0
  - tsdownsample=0.1.2=py310hcb5633a_2
  - typer=0.9.0=pyhd8ed1ab_0
  - types-python-dateutil=2.8.19.20240106=pyhd8ed1ab_0
  - typing-extensions=4.9.0=hd8ed1ab_0
  - typing_extensions=4.9.0=pyha770c72_0
  - typing_utils=0.1.0=pyhd8ed1ab_0
  - tzdata=2023d=h0c530f3_0
  - unicodedata2=15.1.0=py310h2372a71_0
  - uri-template=1.3.0=pyhd8ed1ab_0
  - urllib3=2.1.0=pyhd8ed1ab_0
  - vine=5.1.0=pyhd8ed1ab_0
  - wcslib=8.2.2=h6cde4e1_0
  - wcwidth=0.2.13=pyhd8ed1ab_0
  - webcolors=1.13=pyhd8ed1ab_0
  - webencodings=0.5.1=pyhd8ed1ab_2
  - websocket-client=1.7.0=pyhd8ed1ab_0
  - werkzeug=3.0.1=pyhd8ed1ab_0
  - wheel=0.42.0=pyhd8ed1ab_0
  - widgetsnbextension=4.0.9=pyhd8ed1ab_0
  - wurlitzer=3.0.3=pyhd8ed1ab_0
  - xcb-util=0.4.0=hd590300_1
  - xcb-util-image=0.4.0=h8ee46fc_1
  - xcb-util-keysyms=0.4.0=h8ee46fc_1
  - xcb-util-renderutil=0.3.9=hd590300_1
  - xcb-util-wm=0.4.1=h8ee46fc_1
  - xkeyboard-config=2.40=hd590300_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.1.1=hd590300_0
  - xorg-libsm=1.2.4=h7391055_0
  - xorg-libx11=1.8.7=h8ee46fc_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxrender=0.9.11=hd590300_0
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xf86vidmodeproto=2.3.1=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yaml-cpp=0.8.0=h59595ed_0
  - zeromq=4.3.5=h59595ed_0
  - zfp=1.0.1=h59595ed_0
  - zipp=3.17.0=pyhd8ed1ab_0
  - zlib=1.2.13=hd590300_5
  - zlib-ng=2.0.7=h0b41bf4_0
  - zstandard=0.22.0=py310h1275a96_0
  - zstd=1.5.5=hfc55251_0
  - zziplib=0.13.69=h27826a3_1
  - pip:
      - pipgrip==0.10.12
      - pysm3==3.4.0
      - quaternionarray==0.6.2
      - sotodlib==0+unknown
prefix: /home/tux/miniforge3/envs/soconda-default_0.0.5.dev141

Bai-Chiang commented 10 months ago

Now I could confirm this bug also exist on current main branch.

Here show the changes relative to main branch. After install soconda run run_tests.sh should be able to reproduce error.

diff --git a/config/common.txt b/config/common.txt
index 9d80cc2..6ac10eb 100644
--- a/config/common.txt
+++ b/config/common.txt
@@ -66,4 +66,4 @@ future
 pyaml
 tqdm
 ipykernel
-pyfftw
+pyfftw=0.12
diff --git a/config/default/packages_conda.txt b/config/default/packages_conda.txt
index 2a8b83d..48fc35e 100644
--- a/config/default/packages_conda.txt
+++ b/config/default/packages_conda.txt
@@ -1,5 +1,5 @@
 # Adjust the default python version
-python=3.11.*
+python=3.10.*
 #
 jupyterlab
 wurlitzer
diff --git a/config/default/post_install.sh b/config/default/post_install.sh
index 5b3bcce..f640d65 100644
--- a/config/default/post_install.sh
+++ b/config/default/post_install.sh
@@ -7,9 +7,9 @@
 # for the environment into a user's home directory.

 # Install a module file?
-install_module=yes
+install_module=no

 # Install jupyter kernel setup script?
-install_jupyter_setup=yes
+install_jupyter_setup=no

 # Add any other shell commands here for this system...
diff --git a/pkgs/pixell/meta.yaml b/pkgs/pixell/meta.yaml
index 34eb13f..1e20fa7 100644
--- a/pkgs/pixell/meta.yaml
+++ b/pkgs/pixell/meta.yaml
@@ -43,7 +43,7 @@ requirements:
     - pyyaml
     - h5py
     - pillow
-    - pyfftw
+    - pyfftw=0.12
     # Remove until this perl bug is fixed
     # https://github.com/conda-forge/perl-feedstock/issues/56
     - libxcrypt1   # [linux]
diff --git a/run_tests.sh b/run_tests.sh
index 3dd7c71..65ff175 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -1,10 +1,10 @@
 #!/bin/bash

 # Pixell Tests
-OMP_NUM_THREADS=4 test_pixell.py
+#OMP_NUM_THREADS=4 test_pixell.py

 # Serial TOAST Tests
-OMP_NUM_THREADS=4 MPI_DISABLED=1 python -c 'import toast.tests; toast.tests.run()'
+#OMP_NUM_THREADS=4 MPI_DISABLED=1 python -c 'import toast.tests; toast.tests.run()'

 # MPI-enabled tests
 OMP_NUM_THREADS=2 mpirun -np 4 python -c 'import toast.tests; toast.tests.run()'

tskisner commented 10 months ago

Thanks, I'll try to look into this very soon. Note that pyfftw-0.12 is too old to support python > 3.10. Unfortunately the PR with the fix in the conda-forge feedstock seems to be stalled. In main I lifted the pinning in the hope that the upstream PR would be merged, but I guess that was optimistic...

Bai-Chiang commented 9 months ago

I fix the bug. It was due to the toast meta.ym was point to this commit not the release tag 3.0.0a20.

Bai-Chiang commented 9 months ago

New changes I made:

Updated configuration files. (I'm not sure about site configuration is correct or not. We could change it later.)
I reverted back to bootstrap_base.sh script.
Add .gitignore file, ignoring tmpfs directory and all log files log_* and toast_test_output directory which is created by run_tests.sh.

simonsobs / soconda

Add micromamba support #18

Change log