search

holoviz / datashader

Quickly and accurately render even the largest data.
http://datashader.org
BSD 3-Clause "New" or "Revised" License
3.31k stars 366 forks source link

python 3.11 incompatibility with `getargspec` #1227

Closed stephenholtz closed 1 year ago

stephenholtz commented 1 year ago

OSX 13.4 using a minimal poetry environment in the terminal or vscode:

python = "3.11.*"
numpy = "1.23.4"
datashader = "^0.14.4"
pandas = "^2.0.2"
ipykernel = "^6.23.1"

An error about getargspec prevents execution of the code snippet below which is from the docs:

import pandas as pd
import numpy as np
from collections import OrderedDict as odict

num=10000
np.random.seed(1)

dists = {cat: pd.DataFrame(odict([('x',np.random.normal(x,s,num)), 
                                  ('y',np.random.normal(y,s,num)), 
                                  ('val',val), 
                                  ('cat',cat)]))      
         for x,  y,  s,  val, cat in 
         [(  2,  2, 0.03, 10, "d1"), 
          (  2, -2, 0.10, 20, "d2"), 
          ( -2, -2, 0.50, 30, "d3"), 
          ( -2,  2, 1.00, 40, "d4"), 
          (  0,  0, 3.00, 50, "d5")] }

df = pd.concat(dists,ignore_index=True)
df["cat"]=df["cat"].astype("category")

import datashader as ds
import datashader.transfer_functions as tf

tf.shade(ds.Canvas().points(df,'x','y'))

traceback:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/toolz/functoolz.py:457](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/toolz/functoolz.py:457), in memoize..memof(*args, **kwargs)
    456 try:
--> 457     return cache[k]
    458 except TypeError:

KeyError: (, CPUDispatcher(), CPUDispatcher(), . at 0x136a96d40>, CPUDispatcher(), False)

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/glyph.py:151](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/glyph.py:151), in Glyph._expand_aggs_and_cols(append, ndims, antialiased)
    149 try:
    150     # Numba keeps original function around as append.py_func
--> 151     append_args = inspect.getargspec(append.py_func).args
    152 except (TypeError, AttributeError):
    153     # Treat append as a normal python function

AttributeError: module 'inspect' has no attribute 'getargspec'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
Cell In[2], line 4
      1 import datashader as ds
      2 import datashader.transfer_functions as tf
----> 4 tf.shade(ds.Canvas().points(df,'x','y'))

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/core.py:219](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/core.py:219), in Canvas.points(self, source, x, y, agg, geometry)
    212         raise ValueError(
    213             "source must be an instance of spatialpandas.GeoDataFrame or \n"
    214             "spatialpandas.dask.DaskGeoDataFrame.\n"
    215             "  Received value of type {typ}".format(typ=type(source)))
    217     glyph = MultiPointGeometry(geometry)
--> 219 return bypixel(source, self, glyph, agg)

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/core.py:1260](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/core.py:1260), in bypixel(source, canvas, glyph, agg, antialias)
   1258 with np.warnings.catch_warnings():
   1259     np.warnings.filterwarnings('ignore', r'All-NaN (slice|axis) encountered')
-> 1260     return bypixel.pipeline(source, schema, canvas, glyph, agg, antialias=antialias)

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/utils.py:109](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/utils.py:109), in Dispatcher.__call__(self, head, *rest, **kwargs)
    107 typ = type(head)
    108 if typ in lk:
--> 109     return lk[typ](head, *rest, **kwargs)
    110 for cls in getmro(typ)[1:]:
    111     if cls in lk:

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/data_libraries/pandas.py:17](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/data_libraries/pandas.py:17), in pandas_pipeline(df, schema, canvas, glyph, summary, antialias)
     15 @bypixel.pipeline.register(pd.DataFrame)
     16 def pandas_pipeline(df, schema, canvas, glyph, summary, *, antialias=False):
---> 17     return glyph_dispatch(glyph, df, schema, canvas, summary, antialias=antialias)

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/utils.py:112](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/utils.py:112), in Dispatcher.__call__(self, head, *rest, **kwargs)
    110 for cls in getmro(typ)[1:]:
    111     if cls in lk:
--> 112         return lk[cls](head, *rest, **kwargs)
    113 raise TypeError("No dispatch for {0} type".format(typ))

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/data_libraries/pandas.py:31](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/data_libraries/pandas.py:31), in default(glyph, source, schema, canvas, summary, antialias, cuda)
     29 x_mapper = canvas.x_axis.mapper
     30 y_mapper = canvas.y_axis.mapper
---> 31 extend = glyph._build_extend(x_mapper, y_mapper, info, append, antialias_stage_2)
     33 x_range = canvas.x_range or glyph.compute_x_bounds(source)
     34 y_range = canvas.y_range or glyph.compute_y_bounds(source)

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/toolz/functoolz.py:461](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/toolz/functoolz.py:461), in memoize..memof(*args, **kwargs)
    459     raise TypeError("Arguments to memoized function must be hashable")
    460 except KeyError:
--> 461     cache[k] = result = func(*args, **kwargs)
    462     return result

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/points.py:166](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/points.py:166), in Point._build_extend(self, x_mapper, y_mapper, info, append, antialias_stage_2)
    162 x_name = self.x
    163 y_name = self.y
    165 @ngjit
--> 166 @self.expand_aggs_and_cols(append)
    167 def _perform_extend_points(i, sx, tx, sy, ty, xmin, xmax,
    168                            ymin, ymax, xs, ys, xxmax, yymax,
    169                            *aggs_and_cols):
    170     x = xs[i]
    171     y = ys[i]

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/glyph.py:135](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/glyph.py:135), in Glyph.expand_aggs_and_cols(self, append)
    108 def expand_aggs_and_cols(self, append):
    109     """
    110     Create a decorator that can be used on functions that accept
    111     *aggs_and_cols as a variable length argument. The decorator will
   (...)
    133         Decorator function
    134     """
--> 135     return self._expand_aggs_and_cols(append, self.ndims, self.antialiased)

File [~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/glyph.py:154](https://file+.vscode-resource.vscode-cdn.net/Users/stephenholtz/repos/antenna-analysis/preprocessing/~/Library/Caches/pypoetry/virtualenvs/antenna-analysis-rqYsAEN9-py3.11/lib/python3.11/site-packages/datashader/glyphs/glyph.py:154), in Glyph._expand_aggs_and_cols(append, ndims, antialiased)
    151         append_args = inspect.getargspec(append.py_func).args
    152     except (TypeError, AttributeError):
    153         # Treat append as a normal python function
--> 154         append_args = inspect.getargspec(append).args
    156 # Get number of arguments accepted by append
    157 append_arglen = len(append_args)

AttributeError: module 'inspect' has no attribute 'getargspec'

A simple fix may be to just replace getargspec with getfullargspec which I believe is the suggested drop in replacement.

e.g. in glyph.py 147:154. This ran without error.

        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            try:
                # Numba keeps original function around as append.py_func
                append_args = inspect.getfullargspec(append.py_func).args
            except (TypeError, AttributeError):
                # Treat append as a normal python function
                append_args = inspect.getfullargspec(append).args

I can try to fix this in a PR within a few days unless someone else prefers to do so. A temporary alternative is to monkeypatch at the top of my scripts or on import.

ianthomas23 commented 1 year ago

This is already fixed by #1205. Please search for an existing issues and PRs before creating a new issue.

stephenholtz commented 1 year ago

Ah, apologies. I had not dug deep enough. Thanks for the help

ianthomas23 commented 1 year ago

You're welcome. Version 0.15.0 is now out on PyPI and conda-forge, so you should be able to upgrade.