The creation process for mosaic COGs errors for some volumes. This happens during the step that warps each individual sheet to VRT with a cutline, and we first encountered it on three different specific layers in two different volumes. I'm writing this ticket to record what I found had caused the problem, and the various steps that should be taken to make sure it doesn't happen down the road.
The error in the console was
ERROR 2024-04-05 22:56:33,929 django.contrib.gis err_handler 708714 GDAL_ERROR 1
: b'missing ['
Traceback (most recent call last):
...etc...
File "ohmg/ohmg/loc_insurancemaps/management/commands/volume.py", line 222, in handle
map.generate_mosaic_cog()
File "ohmg/content/models.py", line 167, in generate_mosaic_cog
mosaic_vrt = self.generate_mosaic_vrt()
File "ohmg/content/models.py", line 115, in generate_mosaic_vrt
in_path = latest_sesh.run(return_vrt=True)
File "ohmg/georeference/models/sessions.py", line 476, in run
g = Georeferencer(
File "ohmg/georeference/georeferencer.py", line 157, in __init__
raise TypeError(msg)
TypeError: ERROR: invalid transformation, must be one of dict_keys(['tps', 'poly', 'poly1', 'poly2', 'poly3'])
Ostensibly, an invalid transformation had been passed to the warp command. It turned out that this is because the VRT creation during this process is performed by re-running the latest session that is attached to the resource. This is necessary because the warped VRT is not actually stored on disk, even though it's the exact same one that is used for the georeferencing live preview, so it must be regenerated at this point (doing this only takes a second).
The problem was that for these particular layers, there was an extra, blank georeferencing session that had apparently been created before the preparation session. This shouldn't be allowed to happen, and the UI shouldn't provide any paths to allow it, but if a logged in user went straight to /georeference/<document id> then a new session will actually be created, even if the document has not yet been prepared. In the resource overview, it looks like this:
Looking in the Django admin at this session, it's all completely blank, as you would expect. The deleted_expired_sessions task that runs via celery every minute should theoretically remove this orphan session, but in reality, the way it is implemented via the lock_details on the resource seemed not to trigger the removal of this session.
So, during the COG generation when the "latest" session is run, this blank session gets used, and because it has no information in it, the first error to get thrown is a lack of transformation method (if this didn't error, then plenty of other things would have). Of course, the "latest" session should be the most recent valid session, but After deleting this blank session, the process runs properly without error.
This issue reveals many different places where improvements should be made.
When the mosaic process rewarps each layer to apply a cutline, it should do so using the canonical GCPs, not the latest session
This process should also continue without breaking, even if the warping errors for a single layer
When a user goes to the georeference interface before a document has been prepared, then no georef session should be created
delete_expired_sessions should more thoroughly remove old sessions like these, even though they are not reflected in any particular resource's lock_details
To Explore
Things to explore while addressing this ticket:
What happens when you go straight to /georeference/<document id> before it has been prepared?
How can delete_expired_sessions be improved to take care of these orphans better?
Generally: Better logging throughout the mosaic COG creation process
Can the Georeferencer class be augmented to use a cutline, so that more GDAL operations are pushed from the mosaic creation process to that class?
Describe the problem
The creation process for mosaic COGs errors for some volumes. This happens during the step that warps each individual sheet to VRT with a cutline, and we first encountered it on three different specific layers in two different volumes. I'm writing this ticket to record what I found had caused the problem, and the various steps that should be taken to make sure it doesn't happen down the road.
The error in the console was
Ostensibly, an invalid transformation had been passed to the warp command. It turned out that this is because the VRT creation during this process is performed by re-running the latest session that is attached to the resource. This is necessary because the warped VRT is not actually stored on disk, even though it's the exact same one that is used for the georeferencing live preview, so it must be regenerated at this point (doing this only takes a second).
The problem was that for these particular layers, there was an extra, blank georeferencing session that had apparently been created before the preparation session. This shouldn't be allowed to happen, and the UI shouldn't provide any paths to allow it, but if a logged in user went straight to
/georeference/<document id>then a new session will actually be created, even if the document has not yet been prepared. In the resource overview, it looks like this:Looking in the Django admin at this session, it's all completely blank, as you would expect. The
deleted_expired_sessionstask that runs via celery every minute should theoretically remove this orphan session, but in reality, the way it is implemented via thelock_detailson the resource seemed not to trigger the removal of this session.So, during the COG generation when the "latest" session is run, this blank session gets used, and because it has no information in it, the first error to get thrown is a lack of transformation method (if this didn't error, then plenty of other things would have). Of course, the "latest" session should be the most recent valid session, but After deleting this blank session, the process runs properly without error.
This issue reveals many different places where improvements should be made.
delete_expired_sessionsshould more thoroughly remove old sessions like these, even though they are not reflected in any particular resource'slock_detailsTo Explore
Things to explore while addressing this ticket:
/georeference/<document id>before it has been prepared?Georeferencerclass be augmented to use a cutline, so that more GDAL operations are pushed from the mosaic creation process to that class?