adjust / parquet_fdw

Parquet foreign data wrapper for PostgreSQL
PostgreSQL License
333 stars 37 forks source link

Error on importing foreign schema when columns has white spaces #61

Closed j4david closed 1 year ago

j4david commented 1 year ago

Hello,

I'm getting an error when I import a foreign schema when there parquet files with whitespaces in their column names.

Example:

>>> import pandas as pd;
>>> df = pd.DataFrame(data={'col 1': [1, 2], 'col 2': [3, 4]})
>>> df
   col 1  col 2
0      1      3
1      2      4
>>> df.to_parquet("df.parquet")

create database db1;
postgres=# \c db1

db1=# create extension parquet_fdw;
CREATE EXTENSION

db1=# \dx
                     List of installed extensions
    Name     | Version |   Schema   |           Description
-------------+---------+------------+----------------------------------
 parquet_fdw | 0.2     | public     | foreign-data wrapper for parquet
 plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

db1=# create server parquet_srv foreign data wrapper parquet_fdw;
CREATE SERVER

db1=# import foreign schema "/path/df" from server parquet_srv into public;
ERROR:  syntax error at or near "1"
LINE 1: CREATE FOREIGN TABLE public.df (col 1 bigint, col 2 bigint) ...
                                            ^
QUERY:  CREATE FOREIGN TABLE public.df (col 1 bigint, col 2 bigint) SERVER parquet_srv OPTIONS (filename '/path/df/df.parquet')

When I create foreign table individualy, it work okey:

db1=# create foreign table df_ft ("col 2" int) server parquet_srv options (filename '/path/df/df.parquet');
CREATE FOREIGN TABLE
db1=# select * from df_ft;
 col 2
-------
     3
     4
(2 rows)

Thanks

za-arthur commented 1 year ago

Hi @j4david, Thank you for reporting the issue. I merged the PR #63 which should fix it. Unfortunately parquet_fdw still won't support column names with white spaces fully because of sorted option. sorted option accepts space separated list of columns and therefore it would be hard to fix this without breaking backward compatibility.