cityofaustin / atd-data-tech

Austin Transportation Data & Technology Services
17 stars 2 forks source link

[Tech Debt] Normalize ems__incidents table columns #14316

Open mddilley opened 1 year ago

mddilley commented 1 year ago

Migrating the triggers in this table to the LDM raised some concern about denormalized columns like latitude, longitude, and geometry (which holds a point comprised of the lat/lon). It would be great to research how and why these columns exist and if we can update the ETL, trigger, and table to have fewer columns. We could also use generated columns over triggers if needed. See https://github.com/cityofaustin/atd-vz-data/pull/1308 and the snippet below for more affected columns.

We should check on what columns may exist in end user and app queries before making changes here.

apd_incident_number_1 = ems__incidents.apd_incident_numbers[1],
apd_incident_number_2 = ems__incidents.apd_incident_numbers[2],
mvc_form_date = date(ems__incidents.mvc_form_extrication_datetime),
mvc_form_time = ems__incidents.mvc_form_extrication_datetime::time
frankhereford commented 1 year ago

I think this is a great idea. I would love to see us rely on the geometry column to store the geometry and to access intrinsic data from that geometry through postGIS functions, like st_x(), st_centroid(), etc. This same idea can be applied to our crashes and afd data as well.

patrickm02L commented 12 months ago

11/22

In Refinement, we discussed the need to ask the VZ team, and makes sure that we know what columns would be missed on their end.

Also, created a related Issue #14719 to get to the larger denormalizing of lat and long.