search

citusdata / pg_shard

ATTENTION: pg_shard is superseded by Citus, its more powerful replacement
https://github.com/citusdata/citus
GNU Lesser General Public License v3.0
1.06k stars 63 forks source link

Support UNLOGGED tables #126

Open ergo70 opened 9 years ago

ergo70 commented 9 years ago

Could master_create_worker_shards() be extended in order to optionally create UNLOGGED shard placements?

It seems to work when I change them manually but this is tedious. :smile: But it makes a big difference in write speed. Since there are replicas, the risk of UNLOGGED seems to be lower than in a single node database.

jasonmp85 commented 9 years ago

@ergo70: Are you certain you cannot achieve your performance goals through other means? We've seen pretty high insert rates without UNLOGGED by appropriately tuning PostgreSQL settings. I believe UNLOGGED tables are entirely truncated after a server crash, so your data loss would be total without backups (i.e. you'd lose all data in the table, not just data since some point in time).

One bad effect could occur if you execute a SELECT statement that somehow crashes a worker entirely (say, by executing buggy/unsafe extension code). pg_shard would realize the first replica failed to return a result, so it would continue to the second replica. If the bad code is written such that this query always crashes the server, the replica will crash too. You'd lose both replicas of your data just via that read operation. Presumably all of the other shards for this table would also be UNLOGGED, so any other shards on those two workers would also be truncated. This is quite an effect for a SELECT. I'm not saying none of our users are sophisticated enough to be able to responsibly make this choice, but I do think it might need some special treatment, or at least warnings.

The other side of the coin is that this is a PostgreSQL feature and so it's confusing to not support it. All my concerns above would also exist in a single-node setup and PostgreSQL has happily provided UNLOGGED for some time.

What precisely are your write throughput needs (rows/second)? I should be possible to hit them without UNLOGGED, but if that's absolutely impossible we may need to think outside the box.

ozgune commented 9 years ago

I'm interested in this feature as well. That way, we could have replicated "in-memory tables" for Postgres. :)