Gracefully handle missing project files (ref: `/media/*.qgs` files)

Raruto commented 1 year ago

Checklist

[X] I've searched through the existing issues and this bug has never been reported before

Subject of the issue

Consider sending a more meaningful error message to the client when for some strange reason a user tries to access a project (whose *.qgs) no longer exists on the server.

Right now, you get a generic server error message at client side:

Steps to reproduce

create a new project and save it as usual (eg. uploading the buildings_legend-feature-count.qgs file through the integrated file manager)
delete the uploaded file via docker terminal or sftp: /shared-volume/media/projects/buildings_legend-feature-count.qgs
now go back to the project page to get the following:

Environment

g3w-admin: 3.6.dev-20230503103435
g3w-client: 3.8.0-alpha.2
browser: Firefox 112.0
operating system: Windows 11 64-bit

Link to your project

http://localhost:8000/en/admin/qdjango/buildings/projects/update/public-building-management-demo/

Additional info

Docker compose log

here's what you get when you request the following page at client-side: http://localhost:8000/en/map/buildings/qdjango/1/

2023-05-03 14:30:08 WARNING:root:Serializer
2023-05-03 14:30:08 WARNING:root:Before reading project
2023-05-03 14:30:08 ERROR:qgis.server:[Server] - Error when loading project file '/shared-volume/media/projects/buildings_public-building-management-demo.qgs': Unable to open /shared-volume/media/projects/buildings_public-building-management-demo.qgs 
2023-05-03 14:30:08 WARNING:root:Got project: 
2023-05-03 14:30:08 ERROR:qgis.server:[Server] - Error when loading project file '/shared-volume/media/projects/buildings_public-building-management-demo.qgs': Unable to open /shared-volume/media/projects/buildings_public-building-management-demo.qgs 
2023-05-03 14:30:09 WARNING:qdjango.models.projects:Cannot retrieve QgsMapLayer for QDjango layer buildings_2f43dc1d_6725_42d2_a09b_dd446220104a
2023-05-03 14:30:09 Internal Server Error: /api/config/1/qdjango/1
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 480, in to_representation
2023-05-03 14:30:09     readLeaf(l, ret['layerstree'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 416, in readLeaf
2023-05-03 14:30:09     readLeaf(node, layer['nodes'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 437, in readLeaf
2023-05-03 14:30:09     layer_serialized_data = layer_serialized.data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 743, in to_representation
2023-05-03 14:30:09     qgs_maplayer = self.qgs_project.mapLayers()[instance.qgs_layer_id]
2023-05-03 14:30:09 KeyError: 'buildings_2f43dc1d_6725_42d2_a09b_dd446220104a'
2023-05-03 14:30:09 
2023-05-03 14:30:09 During handling of the above exception, another exception occurred:
2023-05-03 14:30:09 
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
2023-05-03 14:30:09     response = get_response(request)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
2023-05-03 14:30:09     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
2023-05-03 14:30:09     return view_func(*args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/generic/base.py", line 70, in view
2023-05-03 14:30:09     return self.dispatch(request, *args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 505, in dispatch
2023-05-03 14:30:09     response = self.handle_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 465, in handle_exception
2023-05-03 14:30:09     self.raise_uncaught_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
2023-05-03 14:30:09     raise exc
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 502, in dispatch
2023-05-03 14:30:09     response = handler(request, *args, **kwargs)
2023-05-03 14:30:09   File "/code/g3w-admin/client/api/views.py", line 50, in get
2023-05-03 14:30:09     elif 'onlineresource' in ps.data['metadata'] and \
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 483, in to_representation
2023-05-03 14:30:09     'Layer %s is missing from QGIS project!' % l['id'])
2023-05-03 14:30:09 KeyError: 'id'
2023-05-03 14:30:09 Internal Server Error: /api/config/1/qdjango/1
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 480, in to_representation
2023-05-03 14:30:09     readLeaf(l, ret['layerstree'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 416, in readLeaf
2023-05-03 14:30:09     readLeaf(node, layer['nodes'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 437, in readLeaf
2023-05-03 14:30:09     layer_serialized_data = layer_serialized.data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 743, in to_representation
2023-05-03 14:30:09     qgs_maplayer = self.qgs_project.mapLayers()[instance.qgs_layer_id]
2023-05-03 14:30:09 KeyError: 'buildings_2f43dc1d_6725_42d2_a09b_dd446220104a'
2023-05-03 14:30:09 
2023-05-03 14:30:09 During handling of the above exception, another exception occurred:
2023-05-03 14:30:09 
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
2023-05-03 14:30:09     response = get_response(request)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
2023-05-03 14:30:09     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
2023-05-03 14:30:09     return view_func(*args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/generic/base.py", line 70, in view
2023-05-03 14:30:09     return self.dispatch(request, *args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 505, in dispatch
2023-05-03 14:30:09     response = self.handle_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 465, in handle_exception
2023-05-03 14:30:09     self.raise_uncaught_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
2023-05-03 14:30:09     raise exc
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 502, in dispatch
2023-05-03 14:30:09     response = handler(request, *args, **kwargs)
2023-05-03 14:30:09   File "/code/g3w-admin/client/api/views.py", line 50, in get
2023-05-03 14:30:09     elif 'onlineresource' in ps.data['metadata'] and \
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 483, in to_representation
2023-05-03 14:30:09     'Layer %s is missing from QGIS project!' % l['id'])
2023-05-03 14:30:09 KeyError: 'id'
2023-05-03 14:30:09 ERROR:django.request:Internal Server Error: /api/config/1/qdjango/1
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 480, in to_representation
2023-05-03 14:30:09     readLeaf(l, ret['layerstree'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 416, in readLeaf
2023-05-03 14:30:09     readLeaf(node, layer['nodes'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 437, in readLeaf
2023-05-03 14:30:09     layer_serialized_data = layer_serialized.data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 743, in to_representation
2023-05-03 14:30:09     qgs_maplayer = self.qgs_project.mapLayers()[instance.qgs_layer_id]
2023-05-03 14:30:09 KeyError: 'buildings_2f43dc1d_6725_42d2_a09b_dd446220104a'
2023-05-03 14:30:09 
2023-05-03 14:30:09 During handling of the above exception, another exception occurred:
2023-05-03 14:30:09 
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
2023-05-03 14:30:09     response = get_response(request)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
2023-05-03 14:30:09     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
2023-05-03 14:30:09     return view_func(*args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/generic/base.py", line 70, in view
2023-05-03 14:30:09     return self.dispatch(request, *args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 505, in dispatch
2023-05-03 14:30:09     response = self.handle_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 465, in handle_exception
2023-05-03 14:30:09     self.raise_uncaught_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
2023-05-03 14:30:09     raise exc
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 502, in dispatch
2023-05-03 14:30:09     response = handler(request, *args, **kwargs)
2023-05-03 14:30:09   File "/code/g3w-admin/client/api/views.py", line 50, in get
2023-05-03 14:30:09     elif 'onlineresource' in ps.data['metadata'] and \
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 483, in to_representation
2023-05-03 14:30:09     'Layer %s is missing from QGIS project!' % l['id'])
2023-05-03 14:30:09 KeyError: 'id'

Raruto commented 1 year ago

Here's what I get instead after clicking on delete link (when trying to upload a new *.qgs file)

2023-05-03 14:56:39 WARNING:root:Could not resolve form field 'delete_url'.
2023-05-03 14:56:39 Traceback (most recent call last):
2023-05-03 14:56:39   File "/usr/local/lib/python3.10/dist-packages/django/forms/forms.py", line 153, in __getitem__
2023-05-03 14:56:39     field = self.fields[name]
2023-05-03 14:56:39 KeyError: 'delete_url'
2023-05-03 14:56:39 
2023-05-03 14:56:39 During handling of the above exception, another exception occurred:
2023-05-03 14:56:39 
2023-05-03 14:56:39 Traceback (most recent call last):
2023-05-03 14:56:39   File "/usr/local/lib/python3.10/dist-packages/crispy_forms/utils.py", line 80, in render_field
2023-05-03 14:56:39     bound_field = form[field]
2023-05-03 14:56:39   File "/usr/local/lib/python3.10/dist-packages/django/forms/forms.py", line 155, in __getitem__
2023-05-03 14:56:39     raise KeyError(
2023-05-03 14:56:39 KeyError: "Key 'delete_url' not found in 'QdjangoProjectForm'. Choices are: authentication_id, authentication_password, authentication_username, autozoom_query, baselayer, context_base_legend, description, editor2_user, editor_user, editor_user_groups, feature_count_wms, form_id, legend_position, multilayer_query, multilayer_querybybbox, multilayer_querybypolygon, propagate_viewers, qgis_file, qgis_file-metadata, qgis_file-uploads, thumbnail, thumbnail-metadata, thumbnail-uploads, title_ur, toc_layers_init_status, toc_tab_default, toc_themes_init_status, upload_url, url_alias, use_map_extent_as_init_extent, viewer_user_groups, viewer_users."

So, now it seems impossible to upload a new file:

wlorenzetti commented 1 year ago

This is an edge case, because it is highly unlikely that a project will be deleted from the media folder. It can only happen by manually deleting the file from the folder

Raruto commented 1 year ago

It can only happen by manually deleting the file from the folder

I've inadvertently deleted the /shared-volume/ folder while migrating from v3.5 to v3.6 and then i tried to restore it but, if I remember correctly, in such cases the only way to fix is to manually re-upload / individually save all the projects via the graphical interface (that means, even if you re-upload the /shared-volume/media/projects/ folder via FTP, the system is no longer able to link these files to your qdjango projects).

BTW, then to make it faster I simply performed a database reset (anyway, at least when a file is missing the system shouldn't crash..)

g3w-suite / g3w-admin