search

jube-home / jube

Jube is an open-source software designed for monitoring transactions and events. It offers a range of powerful features including real-time data wrangling, artificial intelligence, decision making, and case management. Jube's exceptional performance is particularly evident in its application to fraud prevention and abuse detection scenarios.
https://www.jube.io
GNU Affero General Public License v3.0
29 stars 2 forks source link

Partition Strategy in Cache and Archive Tables #10

Open richard-churchman opened 11 months ago

richard-churchman commented 11 months ago

Data that is used for real-time processing is already logically separated from slower-moving data in the Archive, a partition of sorts. However, Jube greatly underutilises the partitioning capabilities of Postgres.

The task is to modify the Cache and Archive tables to have a partition hierarchy as Tenant Registry ID \ Default >>> Model Default >>>> CreatedDate \ Default.

Most ideally some functionality will exist in the system to be to create and prune partitions automatically. For example, the Cache may have one-day partitions up to a maximum of 7 days. The Archive may have monthly partitions up to a maximum of a year. In a similar manner to the index server, this partition management should exist in a thread inside a Jube instance.

richard-churchman commented 6 months ago

It is unlikely that this will be implemented on the Cache table and tending to Date then Hash partition on the Archive table.