search

mapmanager / MapManagerCore

A Python library with the core functionality of MapManager
0 stars 0 forks source link

Error in creating a MultiImageLoader #8

Closed cudmore closed 2 months ago

cudmore commented 3 months ago

Hi @Suhayb-A thanks for all your work in designing and implementing this backend!

I pulled your current s-dev branch and am getting an error in the example notebook?

Not sure if this was an intermediate error, you may have fixed it in your current local s-dev branch. If so, please push your current s-dev branch to GitHub.

Error in running example notebook

Error occurs when creating a MultiImageLoader with

# Create an image loader
loader = MultiImageLoader(
    lineSegments="../data/rr30a_s0u/line_segments.csv",
    points="../data/rr30a_s0u/points.csv")

Error occurs in base.py in the standalone function setColumnTypes().

            df[key] = df[key].astype(
                valueType) if key in df.columns else pd.Series(dtype=valueType)

This stackoverflow question gave me the answer.

If I change it to this, it does not fail

            try:
                df[key] = df[key].astype(
                    valueType) if key in df.columns else pd.Series(dtype=valueType)
            except (TypeError):
                if key in df.columns:
                    df[key] = np.floor(pd.to_numeric(df[key], errors='coerce')).astype('Int64')
                else:
                    df[key] = pd.Series(dtype=valueType)

I am not suggesting you use the try except but would just include the code from the "except" example above as the main code.

I think this is a common issue with basic data-types not being exactly compatible between pure Python, Pandas, and NumPy. Especially in the handling and representation of missing values with NaN.

Here is the traceback I am seeing

Traceback (most recent call last):
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/arrays/integer.py", line 53, in _safe_cast
    return values.astype(dtype, casting="safe", copy=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Cannot cast array data from dtype('float64') to dtype('int64') according to the rule 'safe'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/cudmore/Sites/MapManagerCore/examples/tryImport.py", line 5, in <module>
    loader = MultiImageLoader(
             ^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/Sites/MapManagerCore/mapmanagercore/loader/imageio.py", line 14, in __init__
    super().__init__(lineSegments, points)
  File "/Users/cudmore/Sites/MapManagerCore/mapmanagercore/loader/base.py", line 219, in __init__
    points = setColumnTypes(points, Spine)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/Sites/MapManagerCore/mapmanagercore/loader/base.py", line 196, in setColumnTypes
    df[key] = df[key].astype(
              ^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/generic.py", line 6640, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 430, in astype
    return self.apply(
           ^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 363, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 758, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 237, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 182, in astype_array
    values = _astype_nansafe(values, dtype, copy=copy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 80, in _astype_nansafe
    return dtype.construct_array_type()._from_sequence(arr, dtype=dtype, copy=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/arrays/masked.py", line 152, in _from_sequence
    values, mask = cls._coerce_to_array(scalars, dtype=dtype, copy=copy)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/arrays/numeric.py", line 266, in _coerce_to_array
    values, mask, _, _ = _coerce_to_data_and_mask(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/arrays/numeric.py", line 223, in _coerce_to_data_and_mask
    values = dtype_cls._safe_cast(values, dtype, copy=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cudmore/opt/miniconda3/envs/mmc-env/lib/python3.11/site-packages/pandas/core/arrays/integer.py", line 59, in _safe_cast
    raise TypeError(
TypeError: cannot safely cast non-equivalent float64 to int64