search

GeoNode / geonode-importer

MIT License
3 stars 15 forks source link

[GPKG] "must specify native_crs" #233

Closed etj closed 7 months ago

etj commented 7 months ago

The importer finds 2 layers:

The GUI reports an error: image

Celery log ``` [2024-04-09 15:49:29,459: INFO/ForkPoolWorker-8] STARTING NEXT STEP importer.import_resource [2024-04-09 15:49:35,113: INFO/ForkPoolWorker-8] Total number of layers available: 2 [2024-04-09 15:49:36,808: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.publish_resource [2024-04-09 15:49:36,808: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.publish_resource for resource: layer_styles, alternate layer_styles [2024-04-09 15:49:37,058: ERROR/ForkPoolWorker-12] error during publishing: must specify native_crs [2024-04-09 15:49:37,146: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.rollback [2024-04-09 15:49:37,146: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.rollback for resource: layer_styles, alternate layer_styles [2024-04-09 15:49:37,150: INFO/ForkPoolWorker-2] Calling rollback for execution_id b98fc0b8-cd6d-481e-90ec-398bee9ed5ad in progress [2024-04-09 15:49:37,264: WARNING/ForkPoolWorker-2] Starting rollback for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles [2024-04-09 15:49:37,264: INFO/ForkPoolWorker-2] Rollback publishing step in progress for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles [2024-04-09 15:49:37,320: ERROR/ForkPoolWorker-12] Task FAILED with ID: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad, reason: must specify native_crs. Request: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad [2024-04-09 15:49:37,361: INFO/ForkPoolWorker-2] Rollback dynamic model & ogr2ogr step in progress for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles [2024-04-09 15:49:37,361: INFO/ForkPoolWorker-2] Dynamic model does not exists, removing ogr2ogr table in progress [2024-04-09 15:49:38,101: WARNING/ForkPoolWorker-2] Rollback for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles completed [2024-04-09 15:49:38,135: ERROR/ForkPoolWorker-12] Task importer.publish_resource[1362c5ff-2480-41aa-9418-d6b704b976f4] raised unexpected: PublishResourceException() Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/importer/celery_tasks.py", line 236, in publish_resource _publisher.publish_resources(data) File "/usr/local/lib/python3.10/dist-packages/importer/publisher.py", line 67, in publish_resources result = self.handler.publish_resources( File "/usr/local/lib/python3.10/dist-packages/importer/handlers/common/vector.py", line 138, in publish_resources raise e File "/usr/local/lib/python3.10/dist-packages/importer/handlers/common/vector.py", line 126, in publish_resources catalog.publish_featuretype( File "/usr/local/lib/python3.10/dist-packages/geoserver/catalog.py", line 898, in publish_featuretype raise ValueError("must specify native_crs") ValueError: must specify native_crs During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/importer/celery_tasks.py", line 282, in publish_resource raise PublishResourceException(detail=error_handler(e, execution_id)) importer.api.exception.PublishResourceException: must specify native_crs. Request: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad [2024-04-09 15:49:41,878: INFO/ForkPoolWorker-12] STARTING NEXT STEP importer.publish_resource [2024-04-09 15:49:41,878: INFO/ForkPoolWorker-12] STARTING NEXT STEP importer.publish_resource for resource: aree, alternate aree [2024-04-09 15:49:44,140: INFO/ForkPoolWorker-9] STARTING NEXT STEP importer.create_geonode_resource [2024-04-09 15:49:44,140: INFO/ForkPoolWorker-9] STARTING NEXT STEP importer.create_geonode_resource for resource: aree, alternate aree [2024-04-09 15:49:44,750: INFO/ForkPoolWorker-9] [2024-04-09 15:49:44,750: INFO/ForkPoolWorker-9] 0 batches, 0 sent [2024-04-09 15:49:44,750: INFO/ForkPoolWorker-9] done in 0.05 seconds [2024-04-09 15:49:50,120: DEBUG/ForkPoolWorker-12] Checking for permissions. [2024-04-09 15:49:50,138: DEBUG/ForkPoolWorker-12] There are no permissions for this object, setting default perms. [2024-04-09 15:49:50,206: DEBUG/ForkPoolWorker-12] Finalizing (permissions and notifications) on resource geonode:aree [2024-04-09 15:49:50,207: DEBUG/ForkPoolWorker-12] Setting permissions {'users': {}, 'groups': {: ['view_resourcebase', 'download_resourcebase']}} on geonode:aree [2024-04-09 15:49:50,524: DEBUG/ForkPoolWorker-12] Removing all permissions on geonode:aree [2024-04-09 15:49:52,463: DEBUG/ForkPoolWorker-12] Adding 0 rule deletion operations for 'geonode:aree [2024-04-09 15:49:52,463: DEBUG/ForkPoolWorker-12] Going to remove 0 rules for layer geonode:aree [2024-04-09 15:49:52,463: DEBUG/ForkPoolWorker-12] Skipping batch execution "Delete geonode:aree" [2024-04-09 15:49:52,467: DEBUG/ForkPoolWorker-12] perm: change_resourcebase_permissions - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,556: DEBUG/ForkPoolWorker-12] perm: view_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,567: DEBUG/ForkPoolWorker-12] perm: change_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,577: DEBUG/ForkPoolWorker-12] perm: change_resourcebase_metadata - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,587: DEBUG/ForkPoolWorker-12] perm: publish_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,598: DEBUG/ForkPoolWorker-12] perm: delete_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,608: DEBUG/ForkPoolWorker-12] perm: download_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,616: DEBUG/ForkPoolWorker-12] perm: view_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,624: DEBUG/ForkPoolWorker-12] perm: download_resourcebase - resource_type: dataset --> assignable: True [2024-04-09 15:49:52,654: INFO/ForkPoolWorker-12] Requesting GeoFence rules on resource "geonode:aree" :: Dataset [2024-04-09 15:49:52,824: DEBUG/ForkPoolWorker-12] Adding 0 rule deletion operations for 'geonode:aree [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:WMS L:aree [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'WMS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 14} [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:GWC L:aree [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'GWC', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 15} [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:WPS L:aree [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'WPS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 16} [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:WFS L:aree [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'WFS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 17} [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:* L:aree [2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 18} [2024-04-09 15:49:52,871: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WMS L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WMS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 19} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:GWC L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'GWC', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 20} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WPS L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WPS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 21} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'request': 'TRANSACTION', 'workspace': 'geonode', 'layer': 'aree', 'access': 'DENY', 'priority': 22} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'request': 'LOCKFEATURE', 'workspace': 'geonode', 'layer': 'aree', 'access': 'DENY', 'priority': 23} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'request': 'GETFEATUREWITHLOCK', 'workspace': 'geonode', 'layer': 'aree', 'access': 'DENY', 'priority': 24} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 25} [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:* L:aree [2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 26} [2024-04-09 15:49:52,950: INFO/ForkPoolWorker-12] Pushing 13 changes into GeoFence for resource aree [2024-04-09 15:49:52,950: DEBUG/ForkPoolWorker-12] Running batch "Set permission for resource geonode:aree" with 13 operations [2024-04-09 15:49:53,191: DEBUG/ForkPoolWorker-12] ... Refresh GeoServer attributes list for Dataset aree [2024-04-09 15:49:53,624: DEBUG/ForkPoolWorker-9] +++++ harvesting_dispatcher starting... +++++ [2024-04-09 15:49:53,640: DEBUG/ForkPoolWorker-9] +++++ harvesting_dispatcher ending... +++++ [2024-04-09 15:49:54,016: DEBUG/ForkPoolWorker-12] Found geoserver resource for this dataset: aree [2024-04-09 15:49:54,609: DEBUG/ForkPoolWorker-12] ... Refresh Legend links for Dataset aree [2024-04-09 15:49:55,236: DEBUG/ForkPoolWorker-12] save_style: Copying style "polygon" to "geonode:aree [2024-04-09 15:49:55,519: DEBUG/ForkPoolWorker-12] set_style: Setting default style "aree" for layer "aree [2024-04-09 15:49:56,083: DEBUG/ForkPoolWorker-12] clean_styles: Retrieving style "aree" for cleanup [2024-04-09 15:49:56,106: DEBUG/ForkPoolWorker-12] clean_styles: Style does not exist: aree [2024-04-09 15:49:56,111: DEBUG/ForkPoolWorker-12] -- Resource Links[Legend link] for layer aree... [2024-04-09 15:49:56,123: DEBUG/ForkPoolWorker-12] -- Resource Links[Legend link]...done! [2024-04-09 15:49:56,204: ERROR/ForkPoolWorker-12] Error 401 reading Style Filter Params GeoWebCache at http://10.10.100.5/geoserver/gwc/rest/layers/geonode:aree.xml [2024-04-09 15:49:56,235: DEBUG/ForkPoolWorker-12] Error 401 invalidating GeoWebCache at http://10.10.100.5/geoserver/gwc/rest/masstruncate [2024-04-09 15:49:56,235: DEBUG/ForkPoolWorker-12] ... Creating Default Resource Links for Dataset aree [2024-04-09 15:49:56,238: DEBUG/ForkPoolWorker-12] -- Resource Links[Compute parameters for the new links]... [2024-04-09 15:49:56,238: DEBUG/ForkPoolWorker-12] -- Resource Links[Set download links for WMS, WCS or WFS and KML]... [2024-04-09 15:49:56,311: DEBUG/ForkPoolWorker-12] -- Resource Links[Set download links for WMS, WCS or WFS and KML]...done! [2024-04-09 15:49:56,311: DEBUG/ForkPoolWorker-12] -- Resource Links[Legend link]... [2024-04-09 15:49:56,320: DEBUG/ForkPoolWorker-12] -- Resource Links[Legend link]...done! [2024-04-09 15:49:56,321: DEBUG/ForkPoolWorker-12] -- Resource Links[OWS Links]... [2024-04-09 15:49:56,334: DEBUG/ForkPoolWorker-12] -- Resource Links[OWS Links]...done! [2024-04-09 15:49:56,335: DEBUG/ForkPoolWorker-12] ... Updating the Catalogue entries for Dataset aree [2024-04-09 15:49:56,582: DEBUG/ForkPoolWorker-12] handling UUID In pre_save_dataset [2024-04-09 15:49:56,589: DEBUG/ForkPoolWorker-12] Checking for permissions. [2024-04-09 15:49:56,787: INFO/ForkPoolWorker-12] Creating style for Dataset geonode:aree / {'sld_file': '', 'sld_uploaded': False, 'vals': {'dirty_state': True}} [2024-04-09 15:49:56,799: INFO/ForkPoolWorker-12] geonode.geoserver.tasks.geoserver_create_style[None]: Using 'MEMCACHED' lock type. [2024-04-09 15:49:56,800: WARNING/ForkPoolWorker-12] geonode.geoserver.tasks.geoserver_create_style[None]: error 3 from memcached_add: CONNECTION FAILURE [2024-04-09 15:49:56,956: DEBUG/ForkPoolWorker-12] 'NoneType' object has no attribute 'href' [2024-04-09 15:49:56,956: DEBUG/ForkPoolWorker-12] 'NoneType' object has no attribute 'href' [2024-04-09 15:49:56,956: DEBUG/ForkPoolWorker-12] 'NoneType' object has no attribute 'href' [2024-04-09 15:49:56,978: DEBUG/ForkPoolWorker-12] Failed to make DELETE request: 403, HTTP Status 403 \u2013 Forbidden

HTTP Status 403 \u2013 Forbidden

Type Status Report

Message Can't delete style referenced by existing layers.

Description The server understood the request but refuses to authorize it.

Apache Tomcat/9.0.85

[2024-04-09 15:49:57,065: DEBUG/ForkPoolWorker-12] save_style: Retrieving style "geonode:aree" for layer "geonode:aree [2024-04-09 15:49:57,222: DEBUG/ForkPoolWorker-12] clean_styles: Retrieving style "aree" for cleanup [2024-04-09 15:49:57,248: DEBUG/ForkPoolWorker-12] clean_styles: Style does not exist: aree [2024-04-09 15:49:57,252: DEBUG/ForkPoolWorker-12] -- Resource Links[Legend link] for layer aree... [2024-04-09 15:49:57,262: DEBUG/ForkPoolWorker-12] -- Resource Links[Legend link]...done! [2024-04-09 15:50:01,410: DEBUG/ForkPoolWorker-12] ...Checking if image is a blank image [2024-04-09 15:50:01,684: ERROR/ForkPoolWorker-9] ['import'] ```

Table gpkg_contents:

table_name  |data_type |identifier  |description|last_change             |min_x            |min_y           |max_x            |max_y     |srs_id|
------------+----------+------------+-----------+------------------------+-----------------+----------------+-----------------+----------+------+
aree        |features  |aree        |           |2024-03-14T13:57:15.624Z|1712986.270199999|5047030.14877623|1786866.833708391|5108676.87|  3003|
layer_styles|attributes|layer_styles|           |2024-03-14T14:00:34.922Z|                 |                |                 |          |     0|

Table layer_styles:

id|f_table_catalog|f_table_schema|f_table_name|f_geometry_column|styleName                                |styleQML  
--+---------------+--------------+------------+-----------------+-----------------------------------------+----------
 1|               |              |aree        |geom             |3classi+urbano+natura+comuni_sempl_140324|<!DOCTYPE  [...]

ogrinfo

ogrinfo ~aap.gpkg 
INFO: Open of `aap.gpkg'
      using driver `GPKG' successful.
1: aree (Multi Polygon)
2: layer_styles (None)
etj commented 7 months ago

An heuristic check to find out if a layer is valid was already in pale, but commented out:

https://github.com/GeoNode/geonode-importer/blob/269ec258b9e3052ac117646e42a038c79ce280be/importer/handlers/common/vector.py#L337

@mattiagiupponi reports that some valid layers were rejected by such check.

We may add a further heuristic check, and avoid ingestion if the geom type is null and the crs is not defined.

mattiagiupponi commented 7 months ago

@etj it was also removed due this PR: https://github.com/GeoNode/geonode-importer/pull/159/ To allow Allow nonspatial layer import. We still make it more smart (at least for the gpkg) and remove the layers we dont want to import