search

electric-sql / electric

Sync little subsets of your Postgres data into local apps and services.
https://electric-sql.com
Apache License 2.0
6.12k stars 143 forks source link

Garbage collect Shapes that are no longer being used #1547

Open balegas opened 3 weeks ago

balegas commented 3 weeks ago

Diamonds are forever, but shapes shouldn't.

Once a shape is created it keeps being updated for eternity, however it might not being consumed anymore because the subscribers are gone, because the origin Postgres no longer exists, etc.

We should have a garbage collection policy in place that ensures that the server stops tracking shapes that are no longer necessary.

kevin-dp commented 3 weeks ago

What if this would be the responsibility of the application? The application is the one that requested the shape in the first place, when it no longer needs a shape, it can send a DELETE request. I'm saying this because we can try to be clever and GC a shape when there are no more subscribers, but an application may unsubscribe from a shape (knowing that it will re-subscribe later) but then when it re-subscribes later it needs to refetch it from scratch because the sync service tried to be clever.

balegas commented 3 weeks ago

I was looking at is more as of a maintenance thing. You'd have a long period before cleaning up a shape. Could also be driven by disk-usage.

KyleAMathews commented 3 weeks ago

Yeah +1 to GC vs explicit maintenance by clients as that's more work to implement. It'd also be hard for an app to know when a shape should be deleted as apps are by definition distributed apps so collecting the state across all the parts of the system on what shapes are in use or not would be quite hard.