adjust / parquet_fdw

Parquet foreign data wrapper for PostgreSQL
PostgreSQL License
351 stars 38 forks source link

Can you reference query arguments and/or filters with files_func? #55

Closed colton-trm closed 1 year ago

colton-trm commented 1 year ago

I've got a use case where I can deterministically identify the parquet file needed by the query, and was trying to figure out how to pass that value to files_func and/or files_func_arg.

In an ideal world, I'd be able to have pseudocode something like this. I've got no difficulty providing a lookup_file_location function that inputs the filters used while accessing the foreign table and outputs filename(s), but I don't see a way to actually get anything about the query except by calling current_query() and doing some messy regex. Any ideas?

CREATE FUNCTION list_parquet_files(args JSONB)
RETURNS TEXT[] AS
$$
BEGIN

    RETURN (SELECT lookup_file_location(args ->> 'query_args'));
    END
$$
LANGUAGE PLPGSQL;

CREATE FOREIGN TABLE foreign_table (one INT8, two INT8[], three TEXT)
SERVER parquet_srv
OPTIONS (
    files_func 'list_parquet_files',
    files_func_arg '{"query_args":query_args}'
);
za-arthur commented 1 year ago

Hi @colton-trm , Unfortunately it isn't possible right now. It might be possible with some significant changes, but we don't have plans to implement that.