Address bit rot

[danielballan]

[Edited]

Summary of the below:

• The solver was taking so long that GH Actions killed it.
• The issue was reproducible locally via jupyter-repo2docker --no-run ..
• Pinning up the dependencies that are known to be challenging to the solver (numpy, hdf5) and removing some dependencies brought the solving time down to about 3 minutes.
• The nsls2-forge channel is no longer needed.
• We removed the source installation of tiled, since that is now on conda-forge.

[danielballan]

  Step 41/54 : RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "binder/environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '
   ---> Running in 2b3cc43312db
  bash: line 1:     8 Killed                  ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "binder/environment.yml"
  time: 4132.264
  Removing intermediate container 2b3cc43312db
  The command '/bin/sh -c TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "binder/environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '' returned a non-zero code: 137

[danielballan]

I confirmed locally with

jupyter-repo2docker .

that the same step hangs for at least one hour:

Step 41/51 : RUN TIMEFORMAT='time: %3R' bash -c 'time mamba env update -p ${NB_PYTHON_PREFIX} -f "binder/environment.yml" && time mamba clean --all -f -y && mamba list -p ${NB_PYTHON_PREFIX} '
 ---> Running in 2da183238069

I suspect GH Actions is killing it for lack of any output. It may be killing it for memory usage (OOM) but it's hard to imagine how mamba could use up all 7 GB of RAM provided to GH Action runners.

[danielballan]

The above was running on a very old version of jupyter-repo2docker, one that pre-dates their switch from conda to micromamba. But now with the latest jupyter-repo2docker (2022.02.0) I still get stuck in the same place. I let it sit for over an hours before killing it:

Step 41/54 : RUN TIMEFORMAT='time: %3R' bash -c 'time ${MAMBA_EXE} env update -p ${NB_PYTHON_PREFIX} --file "binder/environment.yml" && time ${MAMBA_EXE} clean --all -f -y && ${MAMBA_EXE} list -p ${NB_PYTHON_PREFIX} '
 ---> Running in cff2fa5d2444

[danielballan]

Creating the same environment locally, without repo2docker:

$ time micromamba create -f binder/environment.yml  -p from_file --dry-run

finishes in about a minute. I suspect the problem is the two-step creation of the environment --- i.e. it's created with the base requirements for JupyterHub and then updated with environment.yml.

I can extract a Dockerfile like this:

$ jupyter-repo2docker --debug --no-build . > Dockerfile

but it expects a certain execution environment (e.g. supporting files) from repo2docker...this leads me to conclude that path would be fiddly and error-prone.

[danielballan]

Next idea: trying to remove some of the pins from Jupyter-related projects to see if those are creating bad interactions with the base/default environment.

[danielballan]

Update: Nope.

[danielballan]

My current approach is to run

jupyter-repo2docker .

in the repository root, locally, and try commenting out various lines of environment.yml to see which lines (or combinations of lines) create the hang.

[danielballan]

OK, locally https://github.com/NSLS-II/scipy-binder/pull/46/commits/aa3d624053f9692809def31f1851203203f08cfb works.

The only change to I have omitted the python=3.9 pin, so we are getting repo2docker's default of Python 3.7.

It seems that the Python 3.9 pin is incompatible with something. Next step should be to re-pin that and go back through the processing of checking chunks of the requirements to narrow where the issue is.

[danielballan]

A snap shot:

aa3d624 worked in one of the two builds, but it took a very long time.

[danielballan]

Encountered problems while solving:
  - package edrixs-0.0.4-py36hc8ef283_0 requires python >=3.6,<3.7.0a0, but none of the providers can be installed

[danielballan]

For 7241050, the Mamba-reported time (includes solve + download + install) is time: 435.096.

[danielballan]

For 06594dd, the Mamba-reporeted time is time: 666.575.

[danielballan]

The push succeeded: https://hub.docker.com/r/nsls2/scipy-binder/tags