Testing deployment before tag

[tskisner]

Testing install across several systems. Updating github actions.

[tskisner]

• [x] Test build at NERSC, import spt3g, run toast unit tests
• [x] Test jupyter kernel at NERSC
• [x] Test build on local workstation
• [x] Test github CI run on Linux / MacOS
• [x] Test build at site, import spt3g, run MPI toast unit tests
• [x] Test jupyter kernel at site
• [x] Bump versions of toast and sotodlib and re-run CI tests to check user-reported MacOS failures
• [x] Test deploy script at NERSC

[tskisner]

Another note of something I'll fix here: conda-build (the plugin) must now be installed in the base environment along with conda. There is further discussion on these two pages about why this is now needed, due to changes in the conda plugin system:

https://docs.conda.io/projects/conda-build/en/stable/install-conda-build.html https://github.com/conda/conda-build/issues/4995

Note that the compilers package remains installed in the non-base environment. It is only conda plugins (like conda-build) that must be in base.

[tskisner]

Hmmm, actually there are multiple problems, including issues with upstream dependencies, related to python-3.12. Will save those for a future PR.

Another note of something I'll fix here: conda-build (the plugin) must now be installed in the base environment along with conda. There is further discussion on these two pages about why this is now needed, due to changes in the conda plugin system:

https://docs.conda.io/projects/conda-build/en/stable/install-conda-build.html conda/conda-build#4995

Note that the compilers package remains installed in the non-base environment. It is only conda plugins (like conda-build) that must be in base.

[tskisner]

This is now ready for review:

• Reverting bash syntax to work on macos (| tee 2>&1 instead of |& tee)
• Force use of pyfftw wheel until upstream conda package is fixed
• Bump toast version
• Fix comment characters in module init files to match what is used on the system (LMOD vs. TCL modules)
• Add the versioned user directory to the jupyter kernel search path

[Bai-Chiang]

To redirect stderr and stdout to a file only, the old format is correct command >>log 2>&1 or bash specific style command &>>log. The reason of this ordering is explained here.

Also I will make some change for micromamba setup for conda-build changes, since its base env is empty.

[tskisner]

There are more changes coming to this PR (unfortunately). I am debugging segfaults with openmp dependencies (getting closer there), and also dealing with the consequences of conda build now running from the base environment. This impacts things like requiring that conda-build and conda-verify are installed in the base environment, and by default the conda-bld directory in the base environment is used (problematic since it may not be writable). Anyway, sorry for the delay, but wanted you to know I'm not done yet here

[iparask]

@tskisner, there is a new package in sotodlib, numdifftools. See here

[tskisner]

I believe all previous review comments are addressed. The remaining segfault from OpenMP when importing scipy before toast is actually a separate problem that began with the latest packaging of scipy-1.12. I would like to address that in a separate PR since this one has been hanging around too long already. @Bai-Chiang , any further changes needed here before merging?

[Bai-Chiang]

I will make few changes for micromamba, since conda-build is now installed in base environment, this currently breaks micromamba installation.

[tskisner]

Ok, sounds good. In the mean time I am going to setup a new install of this branch at NERSC for testing.