search

citusdata / cstore_fdw

Columnar storage extension for Postgres built as a foreign data wrapper. Check out https://github.com/citusdata/citus for a modernized columnar storage implementation built as a table access method.
Apache License 2.0
1.76k stars 171 forks source link
columnar-storage columnar-store compression postgresql

readme

Important notice: Columnar storage is now part of Citus

Columnar storage is now part of the Citus extension, which uses the table access method API to give a much more native experience. It also supports streaming replication, archival, rollback, and simplifies pg_upgrade. You can use Citus just for columnar storage on a single PostgreSQL server, or combine it with distributed tables to parallelize queries locally or across a cluster of PostgreSQL servers.

Migration is simple, and you'll typically see improved compression thanks to zstd:

-- After adding adding shared_preload_libraries = 'citus'
-- to postgresql.conf and restarting:
CREATE EXTENSION IF NOT EXISTS citus;

-- Create a table using the columnar access method, with the same columns
-- as an existing cstore_fdw table
CREATE TABLE customer_reviews_am (
  LIKE customer_reviews_fdw INCLUDING ALL
) USING columnar;

-- Copy data from an old cstore_fdw table to an access method table
INSERT INTO customer_reviews_am SELECT * FROM customer_reviews_fdw;

-- cstore_fdw data size
SELECT pg_size_pretty(cstore_table_size('customer_reviews_fdw'));
┌────────────────┐
│ pg_size_pretty │
├────────────────┤
│ 100 MB         │
└────────────────┘

-- Citus Columnar data size
SELECT pg_size_pretty(pg_table_size('customer_reviews_am'));
┌────────────────┐
│ pg_size_pretty │
├────────────────┤
│ 64 MB          │
└────────────────┘

Read more about it in the Citus columnar blog post by Jeff Davis.

cstore_fdw

Build Status Coverage

Cstore_fdw is an open source columnar store extension for PostgreSQL. Columnar stores provide notable benefits for analytics use cases where data is loaded in batches. Cstore_fdw’s columnar nature delivers performance by only reading relevant data from disk, and it may compress data 6x-10x to reduce space requirements for data archival.

Cstore_fdw is developed by Citus Data and can be used in combination with Citus, a postgres extension that intelligently distributes your data and queries across many nodes so your database can scale and your queries are fast. If you have any questions about how Citus can help you scale or how to use Citus in combination with cstore_fdw, please let us know.

Join the Mailing List to stay on top of the latest developments for Cstore_fdw.

Introduction

This extension uses a format for its data layout that is inspired by ORC, the Optimized Row Columnar format. Like ORC, the cstore format improves upon RCFile developed at Facebook, and brings the following benefits:

Further, we used the Postgres foreign data wrapper APIs and type representations with this extension. This brings:

Building

cstore_fdw depends on protobuf-c for serializing and deserializing table metadata. So we need to install these packages first:

# Fedora 17+, CentOS, and Amazon Linux
sudo yum install protobuf-c-devel

# Ubuntu 10.4+
sudo apt-get install protobuf-c-compiler
sudo apt-get install libprotobuf-c0-dev

# Ubuntu 18.4+
sudo apt-get install protobuf-c-compiler
sudo apt-get install libprotobuf-c-dev

# Mac OS X
brew install protobuf-c

Note. In CentOS 5, 6, and 7, you may need to install or update EPEL 5, 6, or 7 repositories. See this page for instructions.

Note. In Amazon Linux, the EPEL repository is installed by default, but not enabled. See these instructions for how to enable it.

Once you have protobuf-c installed on your machine, you are ready to build cstore_fdw. For this, you need to include the pg_config directory path in your make command. This path is typically the same as your PostgreSQL installation's bin/ directory path. For example:

PATH=/usr/local/pgsql/bin/:$PATH make
sudo PATH=/usr/local/pgsql/bin/:$PATH make install

Note. cstore_fdw requires PostgreSQL version from 9.3 to 12. It doesn't support earlier versions of PostgreSQL.

Usage

Before using cstore_fdw, you need to add it to shared_preload_libraries in your postgresql.conf and restart Postgres:

shared_preload_libraries = 'cstore_fdw'    # (change requires restart)

The following parameters can be set on a cstore foreign table object.

To load or append data into a cstore table, you have two options:

You can use the ANALYZE command to collect statistics about the table. These statistics help the query planner to help determine the most efficient execution plan for each query.

Note. We currently don't support updating table using DELETE, and UPDATE commands. We also don't support single row inserts.

Updating from earlier versions to 1.7

To update an existing cstore_fdw installation from versions earlier than 1.6 you can take the following steps:

Example

As an example, we demonstrate loading and querying data to/from a column store table from scratch here. Let's start with downloading and decompressing the data files.

wget http://examples.citusdata.com/customer_reviews_1998.csv.gz
wget http://examples.citusdata.com/customer_reviews_1999.csv.gz

gzip -d customer_reviews_1998.csv.gz
gzip -d customer_reviews_1999.csv.gz

Then, let's log into Postgres, and run the following commands to create a column store foreign table:

-- load extension first time after install
CREATE EXTENSION cstore_fdw;

-- create server object
CREATE SERVER cstore_server FOREIGN DATA WRAPPER cstore_fdw;

-- create foreign table
CREATE FOREIGN TABLE customer_reviews
(
    customer_id TEXT,
    review_date DATE,
    review_rating INTEGER,
    review_votes INTEGER,
    review_helpful_votes INTEGER,
    product_id CHAR(10),
    product_title TEXT,
    product_sales_rank BIGINT,
    product_group TEXT,
    product_category TEXT,
    product_subcategory TEXT,
    similar_product_ids CHAR(10)[]
)
SERVER cstore_server
OPTIONS(compression 'pglz');

Next, we load data into the table:

\COPY customer_reviews FROM 'customer_reviews_1998.csv' WITH CSV;
\COPY customer_reviews FROM 'customer_reviews_1999.csv' WITH CSV;

Note. If you are getting ERROR: cannot copy to foreign table "customer_reviews" when trying to run the COPY commands, double check that you have added cstore_fdw to shared_preload_libraries in postgresql.conf and restarted Postgres.

Next, we collect data distribution statistics about the table. This is optional, but usually very helpful:

ANALYZE customer_reviews;

Finally, let's run some example SQL queries on the column store table.

-- Find all reviews a particular customer made on the Dune series in 1998.
SELECT
    customer_id, review_date, review_rating, product_id, product_title
FROM
    customer_reviews
WHERE
    customer_id ='A27T7HVDXA3K2A' AND
    product_title LIKE '%Dune%' AND
    review_date >= '1998-01-01' AND
    review_date <= '1998-12-31';

-- Do we have a correlation between a book's title's length and its review ratings?
SELECT
    width_bucket(length(product_title), 1, 50, 5) title_length_bucket,
    round(avg(review_rating), 2) AS review_average,
    count(*)
FROM
   customer_reviews
WHERE
    product_group = 'Book'
GROUP BY
    title_length_bucket
ORDER BY
    title_length_bucket;

Usage with Citus

The example above illustrated how to load data into a PostgreSQL database running on a single host. However, sometimes your data is too large to analyze effectively on a single host. Citus is a product built by Citus Data that allows you to run a distributed PostgreSQL database to analyze your data using the power of multiple hosts. You can easily install and run other PostgreSQL extensions and foreign data wrappers—including cstore_fdw—alongside Citus.

You can create a cstore_fdw table and distribute it using the create_distributed_table() UDF just like any other table. You can load data using the copy command as you would do in single node PostgreSQL.

Using Skip Indexes

cstore_fdw partitions each column into multiple blocks. Skip indexes store minimum and maximum values for each of these blocks. While scanning the table, if min/max values of the block contradict the WHERE clause, then the block is completely skipped. This way, the query processes less data and hence finishes faster.

To use skip indexes more efficiently, you should load the data after sorting it on a column that is commonly used in the WHERE clause. This ensures that there is a minimum overlap between blocks and the chance of them being skipped is higher.

In practice, the data generally has an inherent dimension (for example a time field) on which it is naturally sorted. Usually, the queries also have a filter clause on that column (for example you want to query only the last week's data), and hence you don't need to sort the data in such cases.

Uninstalling cstore_fdw

Before uninstalling the extension, first you need to drop all the cstore tables:

postgres=# DROP FOREIGN TABLE cstore_table_1;
...
postgres=# DROP FOREIGN TABLE cstore_table_n;

Then, you should drop the cstore server and extension:

postgres=# DROP SERVER cstore_server;
postgres=# DROP EXTENSION cstore_fdw;

cstore_fdw automatically creates some directories inside the PostgreSQL's data directory to store its files. To remove them, you can run:

$ rm -rf $PGDATA/cstore_fdw

Then, you should remove cstore_fdw from shared_preload_libraries in your postgresql.conf:

shared_preload_libraries = ''    # (change requires restart)

Finally, to uninstall the extension you can run the following command in the extension's source code directory. This will clean up all the files copied during the installation:

$ sudo PATH=/usr/local/pgsql/bin/:$PATH make uninstall

Changeset

Version 1.7.0

Version 1.5.1

Version 1.5

Version 1.4.1

Version 1.4

Version 1.3

Version 1.2

Version 1.1

Copyright

Copyright (c) Citus Data, Inc.

This module is free software; you can redistribute it and/or modify it under the Apache v2.0 License.

For all types of questions and comments about the wrapper, please contact us at engage @ citusdata.com.