Open kimadactyl opened 2 months ago
It has a tool to generate migrations to fix this (bundle exec database_consistency -f
) and creates 81 migrations. Running these migrations tho is bringing up errors with the data in the database. This is the first one it hits:
== 20240829173419 ChangePartnersIsAPlaceNullConstraint: migrating =============
-- change_column_null(:partners, :is_a_place, false)
bin/rails aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::NotNullViolation: ERROR: column "is_a_place" contains null values
/Users/kim/git/PlaceCal/db/migrate/20240829173419_change_partners_is_a_place_null_constraint.rb:3:in `change'
Caused by:
ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR: column "is_a_place" contains null values
/Users/kim/git/PlaceCal/db/migrate/20240829173419_change_partners_is_a_place_null_constraint.rb:3:in `change'
Caused by:
PG::NotNullViolation: ERROR: column "is_a_place" contains null values
Do you think it's important to resolve this @katjam?
It looks like about 90% of migrations run - we could potentially just merge those ones for now?
I think yes, we should fix this. Running the good ones sounds like a great idea.
If we leave it, the problem will get more and more difficult to fix. I wonder if we also want to do some kind of manual audit of the production DB and clean up stuff but I appreciate this is a big can of worms.
I think there have been a lot of manual commands run on the DB over the years which has probably left it in a bit of a state.
So the migrations that are failing seem to be a lot of ones where the column state is currently nil
and its a boolean. I think if we fix that it'll fix the underlying db - but migrating all the nils to false on the given example is making half the tests fail 😢
Chucked it on the next milestone anyway. I think we can add a CI step to stop it getting any worse as well.
Only 5 not running actually. Not bad at all.
/Users/kim/git/PlaceCal/db/migrate/20240903131833_change_partners_is_a_place_null_constraint.rb
/Users/kim/git/PlaceCal/db/migrate/20240903131860_add_sites_neighbourhoods_site_id_index.rb
/Users/kim/git/PlaceCal/db/migrate/20240903131872_add_sites_site_admin_index.rb
/Users/kim/git/PlaceCal/db/migrate/20240903131876_change_sites_neighbourhoods_neighbourhood_id_null_constraint.rb
/Users/kim/git/PlaceCal/db/migrate/20240903131905_change_users_role_null_constraint.rb
Rubocop is complaning about some of the migrations being irreversable, though so I can't check it in:
db/migrate/20240903131832_change_partners_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :partners, :id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131836_change_events_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :events, :id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131838_change_calendars_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :calendars, :id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131840_change_addresses_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :addresses, :id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131841_change_users_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :users, :id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131857_change_sites_neighbourhoods_neighbourhood_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :sites_neighbourhoods, :neighbourhood_id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131859_change_sites_neighbourhoods_site_id_to_bigint.rb:5:5: C: Rails/ReversibleMigration: change_column is not reversible.
change_column :sites_neighbourhoods, :site_id, :bigint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131906_remove_index_sites_tags_on_site_id_index.rb:5:5: C: Rails/ReversibleMigration: remove_index(without column) is not reversible.
remove_index nil, name: 'index_sites_tags_on_site_id'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131907_remove_index_service_areas_on_neighbourhood_id_index.rb:5:5: C: Rails/ReversibleMigration: remove_index(without column) is not reversible.
remove_index nil, name: 'index_service_areas_on_neighbourhood_id'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131908_remove_index_organisation_relationships_on_partner_subject_id_index.rb:5:5: C: Rails/ReversibleMigration: remove_index(without column) is not reversible.
remove_index nil, name: 'index_organisation_relationships_on_partner_subject_id'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
db/migrate/20240903131909_remove_index_article_partners_on_article_id_index.rb:5:5: C: Rails/ReversibleMigration: remove_index(without column) is not reversible.
remove_index nil, name: 'index_article_partners_on_article_id'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Do we care about this @katjam? I can rewrite them to be reversible but i dont know what a good use of my time that is? My pref would prob be to just have rubocop ignore the database migrations directory, its autogenerated for the most part and feels like one library fighting another.
I do tend to make reversible migrations so they can be tested, deployed and stepped back incrementally. It's the only way to maintain integrity if something unexpected happens, especially when working across environs with different data sets populating them. Not providing down method relies on assumption that nothing will go wrong and that probably is not a sensible assumption.
Having said that - under our current circumstances with very few users actually depending on the service and nothing critical, it may be a use of our time right now. Though I think increasingly, we do have a responsibility to Trans Dimension and Tipping Point audiences.
Totally fair enough. It prob won't take that long, prob just being lazy here. And you're right if it fucks up when deploying fixing it would be an even bigger headache.
We have about 1,000 visitors a month atm across transdim+placecal so i don't think it's fair to say we have very few users any more as well!
Description
I wondered if there was any missing indexes that might be slowing down the responses and discovered Database Consistency gem.
This is reporting quite a list to address:
Motivation
Database inconsistencies slow down responses and make our database less robust.
Acceptance criteria