ankane / pgslice

Postgres partitioning as easy as pie
MIT License
1.11k stars 67 forks source link

Suggestions/questions #56

Closed fatkodima closed 4 months ago

fatkodima commented 4 months ago

I have a few suggestions and want to verify my understanding of the approaches used by this gem. Questions are related to declarative partitioning.

  1. pgslice assumes that we are partitioning append only tables and none of the existing rows are modified? Because there aren't any update/delete triggers.
  2. When adding partitions using pgslice add_partitions, it adds primary keys per partition with unique values generated per partition. This is problematic, for example, when we try to search the record by id and forgot to use partitioning key in the query. It can return incorrect row. I think we should create a global sequence for the parent table, something like described in https://www.dragonflydb.io/faq/postgresql-partition-auto-increment
  3. In the described sequence of steps we have the following:

    step 5 - pgslice fill <table> step 7 - pgslice swap <table> step 8 - pgslice fill <table> --swapped

I think, the step 8 is not really needed and is dangerous. Imagine after the steps 5 and 7 were completed, we now have a partitioned table swapped in place and new rows were inserted with the next id numbers from what it currently has. What happens if we try to run step 8 and copying rows with already existing ids?

What we should really do, is remove step 8, but suggest people to run step 5 again exactly before the step 7, and in step 7 lock both tables, copy missing rows and do the swap.

ankane commented 4 months ago

Hi @fatkodima,

  1. The fill functionality is designed for append-only tables.
  2. It uses a single sequence (from the original table).
  3. This won't happen due to 2.
fatkodima commented 4 months ago

Thanks for the quick answer! Had you considered it to be not only for append-only tables or prefer for it to be that way? We have a usecase to partition the table where any row can be updated.

ankane commented 4 months ago

Not sure I'd like to include the complexity, but if you put together a general way, feel free to share.