sraoss / pg_ivm

IVM (Incremental View Maintenance) implementation as a PostgreSQL extension
Other
994 stars 27 forks source link

Allow to create unlogged IMMVs #91

Open Jamal-B opened 4 months ago

Jamal-B commented 4 months ago

Description

In some cases (e.g. for very large IMMVs containing volatile data), and provided you are well aware of the risks or drawbacks involved, it may be useful to create an IMMV without writing to PostgreSQL Write-Ahead Logs.

PostgreSQL unlogged tables drawbacks are multiple (not replicated, not backuped, ...) but the main one to keep in mind is that they are automatically truncated after a crash or unclean shutdown. If these conditions meet your requirements, this, as with simple unlogged tables, allows to improve write performance, reduce vacuum impact, and produce fewer WALs (thus reducing network usage, backup size and restoration time).

I hesitated to propose this PR because allowing unlogged IMMVs goes a bit against the fact that PostgreSQL does not allow the creation of unlogged materialized views. But since this feature (based on the fact that IMMVs are, under the hood, classical PostgreSQL tables feeded by triggers) is very useful in our case, I thought that it could also be the same for some other users of this superb extension that is pg_ivm.

Please note that I am not a C developer, and I would obviously be more than happy to consider your comments related to the code... or, of course, the feature itself.

Additionally, I organized the commits based on the project's git history, particularly the Prepare 1.10 commit, without being sure that the PR, if accepted, would merit a version bump. I'll let you judge what's best to do.

Tests

  1. IMMVs are created according to the defined persistence
    
    my_database=# \df create_immv
    /*
                                   List of functions
    Schema   |    Name     | Result data type |        Argument data types        | Type 
    ------------+-------------+------------------+-----------------------------------+------
    pg_catalog | create_immv | bigint           | text, text, boolean DEFAULT false | func
    */

select create_immv('test_immv_1', 'select 1'); select create_immv('test_immv_2', 'select 1', false); select create_immv('test_immv_3', 'select 1', true);

select relname, relpersistence from pg_class where relname ~ 'testimmv';

/ relname | relpersistence -------------+---------------- test_immv_1 | p test_immv_2 | p test_immv_3 | u /


2. IMMVs persistence is preserved after `refresh_immv()` using `with_data = true`
```sql
select refresh_immv('test_immv_1', true);
select refresh_immv('test_immv_2', true);
select refresh_immv('test_immv_3', true);

select relname, relpersistence
from   pg_class
where  relname ~ 'test_immv_';

/*
   relname   | relpersistence 
-------------+----------------
 test_immv_1 | p
 test_immv_2 | p
 test_immv_3 | u
*/
  1. IMMVs persistence is preserved after refresh_immv() using with_data = false
    
    select refresh_immv('test_immv_1', false);
    select refresh_immv('test_immv_2', false);
    select refresh_immv('test_immv_3', false);

select relname, relpersistence from pg_class where relname ~ 'testimmv';

/ relname | relpersistence -------------+---------------- test_immv_1 | p test_immv_2 | p test_immv_3 | u /

Jamal-B commented 3 months ago

I just pushed a rebased (and tested) version of this PR which follows the merge of version 1.9 a few weeks ago.

yugo-n commented 1 month ago

Thank you for your pull request.

Sorry for late reply, but after some consideration, I don't think supporting unlogged IMMVs is a good idea for now for the same reason why unlogged materialized views are disallowed in PostgreSQL.

Unlogged materialized views has been disallowed since the beginning of the matview support in PosgreSQL, i.e. 9.3, by the commit below. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=3223b25ff737c2bf4a642c0deb7be2b30bfecc6e

As explained in the commit message and comments in the code, When a materialized view is made empty after a crash, user cannot distinguish if this has been unpopulated due to the crash or this is actually empty because the system catalog changes (disabling pg_class.relispopulated) are not handled by crash recovery. pg_ivm has the same problem, or It is worse for pg_ivm because IMMV's contents are supposed to be always up-to-date by the immediate maintenance. For this reason, I don't have plan to support unlogged IMMVs for now.