search

citusdata / cstore_fdw

Columnar storage extension for Postgres built as a foreign data wrapper. Check out https://github.com/citusdata/citus for a modernized columnar storage implementation built as a table access method.
Apache License 2.0
1.76k stars 171 forks source link

Check if $PGDATA/cstore_fdw/<oid> folder exists #124

Open gunicsba opened 8 years ago

gunicsba commented 8 years ago

Hello,

I want to have cstore_fdw to part of all the databases I create, so I installed this to template1 database.

When creating a new database it doesn't create a folder under cstore_fdw, and I receive an error: ERROR: could not open file "/data1/pgsql/data/cstore_fdw/2813186/2813202" for writing: No such file or directory

To fix this I created a folder with the oid name of my database, and this fixed the problem.

Can we enhance the fdw to check this?

Also what would be the command I have to execute to create that folder from the database (i.e. using SQL)

Regards, Balazs

mtuncer commented 8 years ago

Hey @Dezzsoke

cstore_fdw creates this folder when you create the server after creating extension.

I was not able to reproduce this issue when I run it locally. Could you provide more context please ?

Regarding of folder creation, AFAIK postgresql does not have build-in support for that. However, you could use plpython to do that. Please see http://stackoverflow.com/questions/34877345/postgresql-trigger-with-python-to-create-folders-based-on-new-field-records

gunicsba commented 8 years ago

To reproduce:

Log into the template1 database. Install cstore_fdw there. Create a server there.

Try to use cstore -> Success.

Create a new database. (it will be based on template1) connect to that database. Try to use cstore -> failure. Create a server -> after this cstore works.

gunicsba commented 8 years ago

I ended up writing a shell script that creates the folder after database creation, this solved the issue. (Not that ugly workaround)

mtuncer commented 7 years ago

We are migrating to postgresql internal files as described in issue #138 .

Work is currently being reviewed. This issue will be resolved with it.

mtuncer commented 7 years ago

@Dezzsoke I am adding a fix that would make sure that directory exists before attempting to create files. However, this still can not address following scenario

I would rather prevent creation of cstore table inside a template database then try to fix this.

gunicsba commented 7 years ago

@mtuncer I think preventing cstore table creation in template databases is acceptable.

mtuncer commented 7 years ago

I am keeping this open due to template issue. cstore should prevent table creation when issued inside a template database.

with 1.6 version, we make sure folder that folder is available before creating a cstore table. We create it if it does not exist.

mtuncer commented 7 years ago

we are moving towards using internal storage where this is no longer an issue