`SystemError` while Importing xESMF

nannau commented 1 year ago

Hi folks! I'm brand new to xESMF. It looks like a great initiative, and I'm looking forward to getting started.

While following the installation instructions with conda (https://xesmf.readthedocs.io/en/latest/installation.html), I ran into a system error when importing xesmf.

The Problem

The error is (full trace below):

import xesmf as xe
...
SystemError: initialization of _internal failed without raising an exception

From an initial glance, this may be a numba issue, not an xESMF issue, but in case it's helpful I thought I'd open it here for other users. See this thread: https://github.com/numba/numba/issues/8615

My System:

OS: Ubuntu 22.04 Conda: 23.1.0 Python: 3.10.9 xesmf: 0.7.0 numpy: 1.24.2

My Solution:

pip install numpy==1.23.5

The above numba GH issue hints that it's related to recent versions of numpy. Indeed, downgrading to numpy == 1.23.5 solves the issue.

Full trace:

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
Cell In[1], line 2
      1 import xarray as xr
----> 2 import xesmf as xe

File ~/miniconda3/envs/xesmf_env/lib/python3.10/site-packages/xesmf/__init__.py:4
      1 # flake8: noqa
      3 from . import data, util
----> 4 from .frontend import Regridder, SpatialAverager
      6 try:
      7     from ._version import __version__

File ~/miniconda3/envs/xesmf_env/lib/python3.10/site-packages/xesmf/frontend.py:13
     10 from xarray import DataArray, Dataset
     12 from .backend import Grid, LocStream, Mesh, add_corner, esmf_regrid_build, esmf_regrid_finalize
---> 13 from .smm import (
     14     _combine_weight_multipoly,
     15     _parse_coords_and_values,
     16     add_nans_to_weights,
     17     apply_weights,
     18     check_shapes,
     19     read_weights,
     20 )
     21 from .util import LAT_CF_ATTRS, LON_CF_ATTRS, split_polygons_and_holes
     23 try:

File ~/miniconda3/envs/xesmf_env/lib/python3.10/site-packages/xesmf/smm.py:7
      4 import warnings
      5 from pathlib import Path
----> 7 import numba as nb
      8 import numpy as np
      9 import sparse as sps

File ~/miniconda3/envs/xesmf_env/lib/python3.10/site-packages/numba/__init__.py:42
     38 from numba.core.decorators import (cfunc, generated_jit, jit, njit, stencil,
     39                                    jit_module)
     41 # Re-export vectorize decorators and the thread layer querying function
---> 42 from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
     43                             get_num_threads, set_num_threads,
     44                             set_parallel_chunksize, get_parallel_chunksize,
     45                             get_thread_id)
     47 # Re-export Numpy helpers
     48 from numba.np.numpy_support import carray, farray, from_dtype

File ~/miniconda3/envs/xesmf_env/lib/python3.10/site-packages/numba/np/ufunc/__init__.py:3
      1 # -*- coding: utf-8 -*-
----> 3 from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
      4 from numba.np.ufunc._internal import PyUFunc_None, PyUFunc_Zero, PyUFunc_One
      5 from numba.np.ufunc import _internal, array_exprs

File ~/miniconda3/envs/xesmf_env/lib/python3.10/site-packages/numba/np/ufunc/decorators.py:3
      1 import inspect
----> 3 from numba.np.ufunc import _internal
      4 from numba.np.ufunc.parallel import ParallelUFuncBuilder, ParallelGUFuncBuilder
      6 from numba.core.registry import DelayedRegistry

SystemError: initialization of _internal failed without raising an exception

Happy to provide more details if necessary.

aulemahal commented 1 year ago

Hi @nannau !

As far as I could see in my quick search, no version of numba has ever supported numpy 1.24 in the last year. I might be wrong, but I think this might be a bug from your package manager? Conda shouldn't have installed that numpy. May be this is related to #185 ? There, the conda recipe for numba was wrong.

In any case, AFAIU this is not an issue of xESMF, but of upstream packages.

nannau commented 1 year ago

Hi there @aulemahal -- thanks for the speedy reply.

As far as I could see in my quick search, no version of numba has ever supported numpy 1.24 in the last year. I might be wrong, but I think this might be a bug from your package manager? Conda shouldn't have installed that numpy.

Interesting! Actually, I just tried again from scratch with: conda install -c conda-forge xesmf and it did install numpy == 1.23.5, so you're absolutely right.

I should mention I deviated slightly from the documentation and installed jupyterlab with pip after conda install -c conda-forge pip. Somehow I must have inadvertently upgraded numpy and angered the dependency gods. I'm trying to reproduce the error at the moment but can't. 🤷🤷🤷

In any case, AFAIU this is not an issue of xESMF, but of upstream packages.

Agreed! This appears to be a weird order-of-operations style issue that's my fault. Closing this issue. Thanks again for your response!

In case someone has this issue in the future, try from scratch following the documentation exactly, or just downgrade numpy. :)

pangeo-data / xESMF