Dedalus issue on Google Colab

[janzika]

Hello Dedalus peeps

I recently took over a GFD course at UNSW, Sydney from Shane Keating which uses Dedalus on Google Colab for some of the computer lab sessions.

We recently encountered an error getting Dedalus to work in our labs.

This is the code used for installation which was working fine until a week or two ago:

## Check Dedalus isn't already installed
try:
  import dedalus.public as de
  print("Dedalus already installed")
except:
  print("Dedalus not installed yet. Let's do it.")

  # Step 1: Install FFTW
  !apt-get install libfftw3-dev
  !apt-get install libfftw3-mpi-dev

  # Step 2: Set paths for Dedalus installation
  import os
  os.environ['MPI_INCLUDE_PATH'] = "/usr/lib/x86_64-linux-gnu/openmpi/include"
  os.environ['MPI_LIBRARY_PATH'] = "/usr/lib/x86_64-linux-gnu"
  os.environ['FFTW_INCLUDE_PATH'] = "/usr/include"
  os.environ['FFTW_LIBRARY_PATH'] = "/usr/lib/x86_64-linux-gnu"

  # Step 3: Install Dedalus using pip
  !echo "Cython<3" > cython_constraint.txt
  !CC=mpicc PIP_CONSTRAINT=cython_constraint.txt pip3 install dedalus==2.1905

  # Step 4: Check Dedalus is importable
  print()
  print()
  try:
    import dedalus.public as de
    print("Dedalus successfully installed :)")
  except:
    print("Error installing Dedalus :(")
    raise

We now get the following error:

Dedalus not installed yet. Let's do it.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libfftw3-dev is already the newest version (3.3.8-2ubuntu8).
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libfftw3-mpi-dev is already the newest version (3.3.8-2ubuntu8).
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.
Collecting dedalus==2.1905
  Using cached dedalus-2.1905.tar.gz (105 kB)
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Error installing Dedalus :(
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-5-5d60e4253bfd>](https://8lbqamp6wig-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in <cell line: 2>()
      2 try:
----> 3   import dedalus.public as de
      4   print("Dedalus already installed")

ModuleNotFoundError: No module named 'dedalus'

During handling of the above exception, another exception occurred:

ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-5-5d60e4253bfd>](https://8lbqamp6wig-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in <cell line: 2>()
     24   print()
     25   try:
---> 26     import dedalus.public as de
     27     print("Dedalus successfully installed :)")
     28   except:

ModuleNotFoundError: No module named 'dedalus'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

Any thoughts on how to fix this issue would be greatly appreciated! Jan

[kburns]

Thanks for the report. It looks like colab is being affected by new versions of mpi4py, which are incompatible with older versions of mpi.

Here's a new installation cell that should get things working on colab, which I've pushed to the notebooks in the cism_dedalus_2023 repository:

# Set environment variables for best performance
%env OMP_NUM_THREADS=1
%env NUMEXPR_MAX_THREADS=1

# Minimize logging output
import logging
logging.disable(logging.DEBUG)

# Check if running on google colab
import os
using_google_colab = bool(os.getenv("COLAB_RELEASE_TAG"))

# Check for Dedalus
try:
    import dedalus.public as de
    print("Dedalus already installed :)")
except:
    print("Dedalus not installed yet.")
    if using_google_colab:
        print("Installing for Google Colab.")
        print()
        # Step 1: Install FFTW
        !apt-get install libfftw3-dev
        !apt-get install libfftw3-mpi-dev
        # Step 2: Set paths for Dedalus installation
        import os
        os.environ['MPI_INCLUDE_PATH'] = "/usr/lib/x86_64-linux-gnu/openmpi/include"
        os.environ['MPI_LIBRARY_PATH'] = "/usr/lib/x86_64-linux-gnu"
        os.environ['FFTW_INCLUDE_PATH'] = "/usr/include"
        os.environ['FFTW_LIBRARY_PATH'] = "/usr/lib/x86_64-linux-gnu"
        # Step 3: Install Dedalus using pip
        !pip3 install cython "mpi4py<4.0" numpy setuptools wheel
        !CC=mpicc pip3 install --no-cache --no-build-isolation http://github.com/dedalusproject/dedalus/zipball/master/
        !pip3 install -q ipympl
        # Step 4: Check installation
        print()
        try:
            import dedalus.public as de
            print("Dedalus successfully installed :)")
        except:
            print("Error installing Dedalus :(")
            raise
    else:
        print("See website for installation instructions:")
        print("https://dedalus-project.readthedocs.io/en/latest/pages/installation.html")

# Setup interactive matplotlib
if using_google_colab:
    from google.colab import output
    output.enable_custom_widget_manager()

[kburns]

The old cython pin should no longer be necessary, we just need to limit mpi4py<4.0.

[janzika]

Thank you Keaton. That has fixed it. You are a legend!