Bump into AttributeError: 'Series' object has no attribute 'is_empty' when trying to display simple things

devleaks commented 8 months ago

Just trying to reproduce demos like airport layout. It seems that airport = Overpass.request(area=dict(iata="BRU"), aeroway=True) does not fetch data. Possible? How can I check? Then, for any simple display, it never goes through

if x.shape[0] > 0 and "geometry" in x.columns:
---> 75             x = x.query("geometry == geometry and not geometry.is_empty")
     76             if x.shape[0] > 0:
     77                 data.loc[x.index, ["latitude", "longitude"]] = x.assign(
     78                     longitude=lambda df: df.geometry.centroid.x,

with above error. I have the feeling data is not fetch. How can I proceed to debug this? Seems an Overpass issue, other osm stuffs go through (Nominatim, for example) Thank you for double checking.

Python 3.10, pip installed traffic, cartes, no more.

devleaks commented 8 months ago

OK, more info after digging a little bit. Data is there. If I replace .is_empty with .notna() we go a few step further but not a lot. It then complains on the type() of data (which has no geometry, etc.) There really is an issue in that get method. Yes, I got it working in the past, but it no longer works. Versions of GeoPandas? Please help, I don't know how to search/solve. Thanks P.

devleaks commented 8 months ago

In desperate move, downgraded to 0.7.5 and there it works like a charm...

xoolive commented 8 months ago

Thanks for the report @devleaks

I cannot reproduce it on my side. Actually, airports are basically my only use case, so I would like to be able to fix things before it backfires to my face.

Could you please run the following, get the name of the cache file, and attach it to the issue?

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> from cartes.osm import Overpass
>>> airport = Overpass.request(area=dict(iata="BRU"), aeroway=True)
INFO:cartes.utils.cache:Looking for cache file: /home/xo/.cache/cartes/osm/a03632683310a42ef441b13959881d1c.json

Additionally, could you please run pip freeze and confirm your geopandas version?

devleaks commented 8 months ago

This is really weird. Here are steps to reproduce in my environment: (MacOS 14.2.1, M1)

conda 24.1.2.
create env traffic python=3.10
pip install traffic ipykernel
run, get error.
downgrade cartes to 0.7.5, 0.7.4, same issue. (did not test 0.7.3...)
downgrade to cartes 0.7.2, no more issue, works perfectly (GeoPandas 0.14.3 all times) pip install -U cartes==0.7.2
REUPGRADE to cartes 0.7.6: No more issue! pip install -U cartes

So downgrading to 0.7.2 installs something that make it work... Enclosed: pip freeze for traffic with cartes 0.7.6 version, and 0.7.2

Script that is run:

from cartes.osm import Overpass
airport = Overpass.request(area=dict(iata="BRU"), aeroway=True)
import matplotlib.pyplot as plt
from matplotlib.offsetbox import AnchoredText

from cartes.crs import Amersfoort, PlateCarree

fig, ax = plt.subplots(
    figsize=(15, 15),
    subplot_kw=dict(projection=Amersfoort())
)

airport.plot(
    ax, by="aeroway",
    gate=dict(color="C0"),  # change default color for gates to C0 blue
    tower=dict(markersize=3000)  # enlarge default size
)
ax.spines["geo"].set_visible(False)

# Focus on the terminal area (gates), with a little buffer (in terms of lat/lon degrees)
ax.set_extent(airport.query('aeroway=="gate"'), buffer=1e-3)

# This is about some semi-automatic placement of text labels

def horizontal(name):
    if name[0] in "BCMH":
        if int(name[-1]) & 1 == 0: return "right"
    if name[0] in "EFG":
        if int(name[-1]) & 1 == 1: return "right"
    return "left"

def vertical(name):
    if name[0] in " D":
        if int(name[-1]) & 1 == 1: return "bottom"
    if name[0] in "MH":
        if int(name[-1]) & 1 == 0: return "bottom"
    return "top"

style = dict(
    transform=PlateCarree(),
    color="white", fontsize=8,
    fontweight="bold", font="Fira Sans",  # what else?
    bbox=dict(facecolor="C0", edgecolor="none", boxstyle="round"),
)

for _, elt in airport.query('aeroway == "gate"').data.iterrows():
    if elt.ref == elt.ref:
        ax.text(
            elt.longitude, elt.latitude, elt.ref.center(3),
            ha=horizontal(elt.ref), va=vertical(elt.ref),
            rotation=45 if elt.ref[0] == "D" else 0, **style
        )

# Title of the map

text = AnchoredText(
    "Brussels airport",
    loc=3,
    prop={"size": 12, "fontname": "Frutiger", },
    frameon=True,
)
text.patch.set_boxstyle("round,pad=0.,rounding_size=0.2")
ax.add_artist(text)

Error when using 0.7.6 the first time:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/_y/82x_5v_d3hl19j62bmn1c09r0000gn/T/ipykernel_598/1936544406.py in ?()
      7     figsize=(15, 15),
      8     subplot_kw=dict(projection=Amersfoort())
      9 )
     10 
---> 11 airport.plot(
     12     ax, by="aeroway",
     13     gate=dict(color="C0"),  # change default color for gates to C0 blue
     14     tower=dict(markersize=3000)  # enlarge default size

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/cartes/osm/overpass/__init__.py in ?(self, ax, by, **kwargs)
    373     def plot(self, ax, by: Optional[str] = None, **kwargs):
    374         if by is None:
    375             return self.data.plot(ax=ax, transform=PlateCarree(), **kwargs)
--> 376         for key, elt in self.data.groupby(by):
    377             if (
    378                 by not in matplotlib_style
    379                 and key not in matplotlib_style

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/cartes/osm/overpass/__init__.py in ?(self, obj, cls)
     71         else:
     72             x = data.assign(latitude=None, longitude=None)
     73 
     74         if x.shape[0] > 0 and "geometry" in x.columns:
---> 75             x = x.query("geometry == geometry and not geometry.is_empty")
     76             if x.shape[0] > 0:
     77                 data.loc[x.index, ["latitude", "longitude"]] = x.assign(
     78                     longitude=lambda df: df.geometry.centroid.x,

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/frame.py in ?(self, expr, inplace, **kwargs)
   4807             msg = f"expr must be a string to be evaluated, {type(expr)} given"
   4808             raise ValueError(msg)
   4809         kwargs["level"] = kwargs.pop("level", 0) + 1
   4810         kwargs["target"] = None
-> 4811         res = self.eval(expr, **kwargs)
   4812 
   4813         try:
   4814             result = self.loc[res]

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/frame.py in ?(self, expr, inplace, **kwargs)
   4933         if "target" not in kwargs:
   4934             kwargs["target"] = self
   4935         kwargs["resolvers"] = tuple(kwargs.get("resolvers", ())) + resolvers
   4936 
-> 4937         return _eval(expr, inplace=inplace, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/eval.py in ?(expr, parser, engine, local_dict, global_dict, resolvers, level, target, inplace)
    332             resolvers=resolvers,
    333             target=target,
    334         )
    335 
--> 336         parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
    337 
    338         if engine == "numexpr" and (
    339             is_extension_array_dtype(parsed_expr.terms.return_type)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, expr, engine, parser, env, level)
    805         self.env = env or Scope(level=level + 1)
    806         self.engine = engine
    807         self.parser = parser
    808         self._visitor = PARSERS[parser](self.env, self.engine, self.parser)
--> 809         self.terms = self.parse()

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self)
    824     def parse(self):
    825         """
    826         Parse an expression.
    827         """
--> 828         return self._visitor.visit(self.expr)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    408                 raise e
    409 
    410         method = f"visit_{type(node).__name__}"
    411         visitor = getattr(self, method)
--> 412         return visitor(node, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    414     def visit_Module(self, node, **kwargs):
    415         if len(node.body) != 1:
    416             raise SyntaxError("only a single expression is allowed")
    417         expr = node.body[0]
--> 418         return self.visit(expr, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    408                 raise e
    409 
    410         method = f"visit_{type(node).__name__}"
    411         visitor = getattr(self, method)
--> 412         return visitor(node, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    420     def visit_Expr(self, node, **kwargs):
--> 421         return self.visit(node.value, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    408                 raise e
    409 
    410         method = f"visit_{type(node).__name__}"
    411         visitor = getattr(self, method)
--> 412         return visitor(node, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    742             op, op_class, lhs, rhs = self._maybe_transform_eq_ne(node, lhs, rhs)
    743             return self._maybe_evaluate_binop(op, node.op, lhs, rhs)
    744 
    745         operands = node.values
--> 746         return reduce(visitor, operands)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(x, y)
    738         def visitor(x, y):
    739             lhs = self._try_visit_binop(x)
--> 740             rhs = self._try_visit_binop(y)
    741 
    742             op, op_class, lhs, rhs = self._maybe_transform_eq_ne(node, lhs, rhs)
    743             return self._maybe_evaluate_binop(op, node.op, lhs, rhs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, bop)
    732     def _try_visit_binop(self, bop):
    733         if isinstance(bop, (Op, Term)):
    734             return bop
--> 735         return self.visit(bop)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    408                 raise e
    409 
    410         method = f"visit_{type(node).__name__}"
    411         visitor = getattr(self, method)
--> 412         return visitor(node, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    539     def visit_UnaryOp(self, node, **kwargs):
    540         op = self.visit(node.op)
--> 541         operand = self.visit(node.operand)
    542         return op(operand)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    408                 raise e
    409 
    410         method = f"visit_{type(node).__name__}"
    411         visitor = getattr(self, method)
--> 412         return visitor(node, **kwargs)

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/computation/expr.py in ?(self, node, **kwargs)
    649             except AttributeError:
    650                 # something like datetime.datetime where scope is overridden
    651                 if isinstance(value, ast.Name) and value.id == attr:
    652                     return resolved
--> 653                 raise
    654 
    655         raise ValueError(f"Invalid Attribute context {type(ctx).__name__}")

/opt/homebrew/Caskroom/miniforge/base/envs/traffic2/lib/python3.10/site-packages/pandas/core/generic.py in ?(self, name)
   6292             and name not in self._accessors
   6293             and self._info_axis._can_hold_identifiers_and_holds_name(name)
   6294         ):
   6295             return self[name]
-> 6296         return object.__getattribute__(self, name)

AttributeError: 'Series' object has no attribute 'is_empty'

freezes.zip

devleaks commented 8 months ago

Between freezes, pandas goes from 1.5.2 in 0.7.2 to 2.2.1 in 0.7.6. After "reupgrading" to cartes==0.7.6, pandas remains 1.5.2. Geopandas does not change in all this process.

devleaks commented 8 months ago

I confirm the following: cartes==0.7.6 works with GeoPandas==0.14.3 and pandas<2 cartes==0.7.6 exhibit the problem mentioned above with GeoPandas==0.14.3 and pandas>2. Offending command is in OverpassDataDescriptor.get(): x.query() does not complete.

       if x.shape[0] > 0 and "geometry" in x.columns:
             x = x.query("geometry == geometry and not geometry.is_empty")
             if x.shape[0] > 0:
                 data.loc[x.index, ["latitude", "longitude"]] = x.assign(
                     longitude=lambda df: df.geometry.centroid.x,

It seems that at that stage, somehow, the geometry column is not (no longer?) recognized as a geometry column but as a regular pandas.Series. Tried to naively force it with .set_geometry() with no success. Curious to see if someone can reproduce. P. If necessary, we can close this issue, since cartes works with pandas<2. Thank you.

xoolive commented 8 months ago

It worked yesterday with one of the first pandas 2, so the critical version is later. I will have a look when I have some time, hopefully next week. Thanks for the heads up

xoolive commented 8 months ago

Bon j'ai résolu le truc, j'ai pas trop compris:

il faut un .set_geometry() pour certaines fonctions
le query('is_empty') ne marche plus, mais il marche en .loc

Normalement c'est bon, je vais faire une release dans la foulée...

devleaks commented 8 months ago

Ah, d'après le message d'erreur, j'avais aussi déduit que le dataframe, ou en tous les cas la série "geometry" n'était pas/plus perçue comme une géométrie, et j'ai aussi essayé de la forcer en géométrie, mais cela n'avait pas résolu le problème. Peut être fallait-il le faire à plusieurs endroits? Enfin, vous avez trouvé, c'est le principal. J'essaierai la version corrigée et fermerai cet "issue". Merci. Pierre

xoolive commented 8 months ago

En fait il y avait deux trucs:

elle est bien perçue comme une geometrie (la série) mais le dataframe cherchait la "géométrie principale";
le .query('geometry.is_empty') ne marche plus, rien à faire (mais je n'ai pas trouvé d'issue ouvert en ce sens...)

Normalement c'est fermé par mon commit, mais je veux bien une confirmation malgré tout :sweat_smile:

xoolive / cartes

Bump into AttributeError: 'Series' object has no attribute 'is_empty' when trying to display simple things #64