Updating a previously `.qgis` project fails silently

Raruto commented 1 year ago

Checklist

[X] I've searched through the existing issues and this bug has never been reported before

Subject of the issue

I'm trying to replace a previously uploaded qgis project with a new one via administrative interface, but once saved it always shows the previous file.

Steps to reproduce

download and unzip G3WSUITE_3.5.zip
extract project_data\spatialite folder and upload it as is to server (eg: into \shared-volume\project_data\spatialite)
extract projects/building-management-demo.qgs and create a new project via g3w-admin interface
save it and re-open the edit page (eg: http://localhost:8000/it/admin/qdjango/buildings/projects/update/public-building-management-demo/)
the displayed .qgis project file name should match the following:

now extract projects/building-management-demo-2.qgs
upload it to the previously created project via g3w-admin interface, saving it and re-opening the edit page (eg: http://localhost:8000/it/admin/qdjango/buildings/projects/update/public-building-management-demo/)
the displayed .qgis project file name still match the previous one:

That is: building-management-demo.qgs instead of building-management-demo-2.qgs

Environment

g3w-admin: 3.5.dev-20230313134606
g3w-client: 3.8.0-alpha.1
browser: Firefox 111.0
operating system: Windows 10 64-bit

Link to your project

http://localhost:8000/it/admin/qdjango/buildings/projects/update/public-building-management-demo/

Additional info

The admin interface does not return any errors (e.g. preventing file saving), below the docker container log at the time of saving:

Logged warning: Loading a file that was saved with a newer version of qgis (saved in 3.26.0-Buenos Aires, loaded in 3.22.16-Białowieża). Problems may occur.
[] - Picture expression eval error
[] - Picture expression eval error
[] - Picture expression eval error
[] - Picture expression eval error
[] - Picture expression eval error
[] - Picture expression eval error
[Server] - Warning, Layer(s) buildings_2f43dc1d_6725_42d2_a09b_dd446220104a, buildings_rating_3d535fae_fd04_4df6_b6ff_8cbd13df078f, maintenance_works_f8cbe34a_eebe_4cd1_9c78_5d420ab0af63, roads_ea006d6f_bd87_4635_aae0_4e9e7842b3f4, type_subtype_caec4a0b_e7c4_4542_b59c_769f2033d6b1, work_areas_f0ecbe28_cbd1_4a38_8a57_ab6da91473fe not valid in project /shared-volume/media/temp_uploads/8be0b0fde1844b1ab3dd10e812c55d99.qgs

Raruto commented 1 year ago

I need to double check, as I noticed that by editing with QGIS a file (ref: building-management-demo.qgs --> building-management-demo-2.qgs ) the paths are now saved in absolute manner:

<Layer provider="spatialite" source="dbname='C:/Users/<USERNAME>/Desktop/G3WSUITE_3.5/project_data/spatialite/building_management_demo.sqlite' ...

wlorenzetti commented 1 year ago

@Raruto the new project replaces the old file name, but the content matches the new one right?

Raruto commented 1 year ago

the new project replaces the old file name, but the content matches the new one right?

This happens with 100% probability:

after replacing building-management-demo.qgs with: building-management-demo-2.qgs within the edit page and you should get:

But, on frontend, I think there could be some other caching issues.

To be clear, this should be the main difference between the two projects (3 layer categories vs 1 layer category):

building-management-demo.qgs

building-management-demo-2.qgs

But after trying to reload the frontend map (several times and after replacing: building-management-demo.qgs with building-management-demo-2.qgs), it happens very often that the project remains in this previouse state, ie:

Once has happened that the classes that should be hidden are still displayed, but greyed out:

The project seems to have loaded correctly only the first time I re-loaded the page:

wlorenzetti commented 1 year ago

This behavior happens to me too some times, I think is related to development environment, we can try settings this django settings to True:

G3WADMIN_USE_CUSTOM_CACHE_INVALIDATOR  = True

Raruto commented 1 year ago

@wlorenzetti I set this in my settings_docker.py,

G3WADMIN_USE_CUSTOM_CACHE_INVALIDATOR  = True

but I can't get these debug lines to printed in the console:

https://github.com/g3w-suite/g3w-admin/blob/c90e51e27b5c0d371b5b716809292fb4790f423c/g3w-admin/qdjango/apps.py#L147-L159

Do I have to change some other setting in here?

# g3w-suite-docker/g3w-suite/settings_docker.py#L133-L208

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        },
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue'
        }
    },
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler',
            'formatter': 'verbose'
        },
        'file': {
            'level': 'ERROR',
            'class': 'logging.FileHandler',
            'filename': '/tmp/error.log',
            'formatter': 'verbose'
        },
        'file_debug': {
            'level': 'DEBUG',
            'filters': ['require_debug_true'],
            'class': 'logging.FileHandler',
            'filename': '/tmp/debug.log',
            'formatter': 'verbose'
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['console', 'mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        'g3wadmin.debug': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
        'pycsw.server': {
            'handlers': ['console'],
            'level': 'ERROR',
        },
        'django.db.backends': {
            'handlers': ['console'],
            'level': 'ERROR',
        },
        'catalog': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
        'celery.task': {
            'handlers': ['console'],
            'level': 'DEBUG',
        },
        'openrouteservice': {
            'handlers': ['console'],
            'level': 'DEBUG',
        }
    }
}

Just to know, because when I use print('some stuff') it should usually works.

Raruto commented 1 year ago

I suspect that ProjectCacheInvalidator and QgsConfigCache classes make use of different nomeclature for stored projects.

https://github.com/g3w-suite/g3w-admin/blob/e9a9916dfcfad1384fb717e76920c67eecde8c23/g3w-admin/qdjango/apps.py#L190-L193

That means, this is how QGIS server currently saves / caches the previously cached project:

# GROUP NAME    = "buildings"
# PROJECT TITLE = "Public Building Management Demo"

/shared-volume/media/projects/buildings_public-building-management-demo.qgs

And here is what happens when you try to replace that project using a different "project title":

And finally, when you visit the map url (eg: http://localhost:8000/it/map/buildings/qdjango/1/):

# GROUP NAME    = "buildings"
# PROJECT TITLE = "Public Building Management Demo 22"

/shared-volume/media/projects/buildings_public-building-management-demo-22.qgs

NB I don't know if it's intented, but as you can see the ProjectCacheInvalidator class also caches the following temporary file:

/shared-volume/media/temp_uploads/b13e074e-2dfa-4d07-b6c8-4586c0959dc6

g3w-suite / g3w-admin