ckan / ckanext-spatial

Geospatial extension for CKAN
http://docs.ckan.org/projects/ckanext-spatial
125 stars 190 forks source link

Fix catch error on index #327

Closed PavloMalko closed 2 weeks ago

PavloMalko commented 3 months ago

When datasets have the wrong geospatial coverage value, they aren’t indexed so users won’t know of them and can’t update/ delete them:

ckan search-index rebuild -ei
2024-03-13 02:12:13,066 INFO  [ckan.cli] Using configuration file /etc/ckan/default/production.ini
2024-03-13 02:12:13,067 INFO  [ckan.config.environment] Loading static files from public-bs3
2024-03-13 02:12:13,267 WARNI [ckan.common] Option ckan.plugins is not declared
2024-03-13 02:12:13,494 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/default/src/ckan/ckan/templates-bs3
2024-03-13 02:12:13,714 WARNI [ckan.common] Option ckanext.xloader.max_content_length is not declared
2024-03-13 02:12:13,714 WARNI [ckan.common] Option ckanext.xloader.max_type_guessing_length is not declared
2024-03-13 02:12:13,714 WARNI [ckan.common] Option ckanext.xloader.max_excerpt_lines is not declared
2024-03-13 02:12:14,704 WARNI [ckan.common] Option lang is not declared
2024-03-13 02:12:14,782 WARNI [ckanext.reclineview.plugin] The Recline-based views are deprecated andwill be removed in future versions
2024-03-13 02:12:14,825 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/default/src/ckan/ckan/templates-bs3
2024-03-13 02:12:14,904 DEBUG [ckanext.harvest.model] Harvest tables defined in memory
2024-03-13 02:12:14,908 DEBUG [ckanext.harvest.model] Harvest tables already exist
2024-03-13 02:12:14,990 WARNI [ckan.common] Option ckanext_jmh.theme_name is not declared
2024-03-13 02:12:15,211 WARNI [ckan.config.middleware.flask_app] Extensions are excluded from CSRF protection! We allow extensions to run without CSRF protection but it will be forced future releases. Read the documentation for more information on how to add CSRF protection to your extension.
2024-03-13 02:12:15,301 INFO  [ckan.lib.search] Rebuilding search index...
2024-03-13 02:12:15,306 INFO  [ckan.lib.search] Rebuilding the whole index...
Indexing dataset 1/32024-03-13 02:12:15,335 ERROR [ckan.lib.search] Error while indexing dataset ffaedeec-6345-4925-920f-da60ef617e48: TypeError("'float' object is not iterable")
2024-03-13 02:12:15,346 ERROR [ckan.lib.search] Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 253, in rebuild
    package_index.update_dict(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 106, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 290, in index_package
    pkg_dict = item.before_dataset_index(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
    pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 178, in index_dataset
    shape = self.shape_from_geometry(geometry)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
    shape = shapely.geometry.shape(geometry)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/geo.py", line 102, in shape
    return Polygon(ob["coordinates"][0], ob["coordinates"][1:])
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 93, in __new__
    coordinates = np.array([_coords(o) for o in coordinates])
TypeError: 'float' object is not iterable
Indexing dataset 2/32024-03-13 02:12:15,357 ERROR [ckan.lib.search] Error while indexing dataset d6ea3142-0746-4b43-8fdf-cdd2a75643f7: TypeError("'float' object is not iterable")
2024-03-13 02:12:15,363 ERROR [ckan.lib.search] Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 253, in rebuild
    package_index.update_dict(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 106, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 290, in index_package
    pkg_dict = item.before_dataset_index(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
    pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 178, in index_dataset
    shape = self.shape_from_geometry(geometry)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
    shape = shapely.geometry.shape(geometry)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/geo.py", line 108, in shape
    return MultiPolygon([[c[0], c[1:]] for c in ob["coordinates"]])
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/multipolygon.py", line 81, in __new__
    p = polygon.Polygon(shell, holes)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 93, in __new__
    coordinates = np.array([_coords(o) for o in coordinates])
TypeError: 'float' object is not iterable
Indexing dataset 3/32024-03-13 02:12:15,376 ERROR [ckan.lib.search] Error while indexing dataset de972abb-6e84-4bcb-a4d2-cf23a7e306b9: TypeError("'float' object is not iterable")
2024-03-13 02:12:15,381 ERROR [ckan.lib.search] Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 253, in rebuild
    package_index.update_dict(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 106, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 290, in index_package
    pkg_dict = item.before_dataset_index(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
    pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 178, in index_dataset
    shape = self.shape_from_geometry(geometry)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
    shape = shapely.geometry.shape(geometry)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/geo.py", line 108, in shape
    return MultiPolygon([[c[0], c[1:]] for c in ob["coordinates"]])
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/multipolygon.py", line 81, in __new__
    p = polygon.Polygon(shell, holes)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 93, in __new__
    coordinates = np.array([_coords(o) for o in coordinates])
TypeError: 'float' object is not iterable
2024-03-13 02:12:15,382 INFO  [ckan.lib.search] Finished rebuilding search index.`