production - phase deletion takes you to error page

[glw]

When deleting a phase it takes you here;

Thought it does say that the phase was deleted successfully its has not been removed.

[smcalilly]

@glw I've not been able to recreate this. Do you know which project this was for?

[glw]

try project "test" - ... /projects/85/ Try to remove the phase. It should take you to that same error page. I just tested again today and got the same result.

[smcalilly]

@glw very helpful — thanks!

[smcalilly]

first, a glimpse at the logs

2022-08-08T15:23:32.133686+00:00 app[web.1]: [2022-08-08 15:23:32 +0000] [20] [DEBUG] POST /projects/phases/delete/126/
2022-08-08T15:23:32.277375+00:00 app[web.1]: Internal Server Error: /projects/phases/delete/126/
2022-08-08T15:23:32.277377+00:00 app[web.1]: Traceback (most recent call last):
2022-08-08T15:23:32.277378+00:00 app[web.1]: File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 242, in _commit
2022-08-08T15:23:32.277378+00:00 app[web.1]: return self.connection.commit()
2022-08-08T15:23:32.277379+00:00 app[web.1]: psycopg2.errors.ForeignKeyViolation: insert or update on table "asset_dashboard_phasezonedistribution" violates foreign key constraint "asset_dashboard_phas_phase_id_535d6e36_fk_asset_das"
2022-08-08T15:23:32.277379+00:00 app[web.1]: DETAIL:  Key (phase_id)=(126) is not present in table "asset_dashboard_phase".
...
2022-08-08T15:23:32.277394+00:00 app[web.1]: django.db.utils.IntegrityError: insert or update on table "asset_dashboard_phasezonedistribution" violates foreign key constraint "asset_dashboard_phas_phase_id_535d6e36_fk_asset_das"
2022-08-08T15:23:32.277394+00:00 app[web.1]: DETAIL:  Key (phase_id)=(126) is not present in table "asset_dashboard_phase".

i think this is some sort of race condition. we calculate the PhaseZoneDistribution values whenever a LocalAsset is created or deleted. by the time Django executes the queries for the PhaseZoneDistribution, the Phase is already deleted. this causes the foreign key constraint, since PhaseZoneDistribution has a fk for Phase.

if this is true, then this would happen only when deleting a phase that has gis assets.

visually:

• local_asset.delete() -> post_delete signal -> calculates the phasezonedistribution -> deletes the phase -> exception
  • during calculates the phasezonedistribution, does the atomic save happen immediately, or is that put into a queue and only runs later (in this case, after the phase is deleted)

some observations:

• this happens because of the LocalAsset signal (i think — fact check/confirm this)
• i dumped the prod db and recreated the bug where project_id = 6
• for some reason, this only happens when deleting the final phase for a project whenever there are no other phases? (or maybe not? need clarity here)

todo:

• create two phases for a project where both phases have GIS assets. this bug should happen for both phases.
• figure out how to handle the phasezonedistribution in the case where there are no phases or gis assets for a project
  • we'll have to rework when that code is called. we calculate the distribution whenever a local asset is created or deleted.

[smcalilly]

here's a gist with logs from stepping through the code with the debugger: https://gist.github.com/smcalilly/e4c198d81fbe3577ba858a44eeee712f