make install failing

[hpolonskaya]

/parquet_fdw$ make install Makefile:16: /usr/lib/postgresql/11/lib/pgxs/src/makefiles/pgxs.mk: No such file or directory make: *** No rule to make target '/usr/lib/postgresql/11/lib/pgxs/src/makefiles/pgxs.mk'. Stop.

[hpolonskaya]

i fixed first error but now i get this ~/parquet_fdw-master$ make install g++ -std=c++11 -O3 -I. -I./ -I/usr/include/postgresql/11/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/mit-krb5 parquet_impl.cpp -c -fPIC -L/usr/lib/x86_64-linux-gnu -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -L/usr/lib/llvm-6.0/lib -L/usr/lib/x86_64-linux-gnu/mit-krb5 -Wl,--as-needed -lpgcommon -lpgport -lpthread -lselinux -lxslt -lxml2 -lpam -lssl -lcrypto -lgssapi_krb5 -lz -ledit -lrt -lcrypt -ldl -lm -o parquet.o parquet_impl.cpp: In constructor ‘ParquetFdwExecutionState::ParquetFdwExecutionState(const char, bool)’: parquet_impl.cpp:187:77: error: invalid new-expression of abstract class type ‘parquet::arrow::FileReader’ parquet::ParquetFileReader::OpenFile(filename, use_mmap))); ^ In file included from parquet_impl.cpp:11:0: /usr/include/parquet/arrow/reader.h:106:22: note: because the following virtual functions are pure within ‘parquet::arrow::FileReader’: class PARQUET_EXPORT FileReader { ^~~~~~ /usr/include/parquet/arrow/reader.h:126:27: note: virtual arrow::Status parquet::arrow::FileReader::GetColumn(int, std::unique_ptr) virtual ::arrow::Status GetColumn(int i, std::unique_ptr out) = 0; ^~~~~ /usr/include/parquet/arrow/reader.h:129:27: note: virtual arrow::Status parquet::arrow::FileReader::GetSchema(std::shared_ptr) virtual ::arrow::Status GetSchema(std::shared_ptr<::arrow::Schema> out) = 0; ^~~~~ /usr/include/parquet/arrow/reader.h:134:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadColumn(int, std::shared_ptr) virtual ::arrow::Status ReadColumn(int i, ^~~~~~ /usr/include/parquet/arrow/reader.h:149:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadSchemaField(int, std::shared_ptr) virtual ::arrow::Status ReadSchemaField( ^~~~~~~ /usr/include/parquet/arrow/reader.h:155:27: note: virtual arrow::Status parquet::arrow::FileReader::GetRecordBatchReader(const std::vector&, std::unique_ptr) virtual ::arrow::Status GetRecordBatchReader( ^~~~~~~~ /usr/include/parquet/arrow/reader.h:174:27: note: virtual arrow::Status parquet::arrow::FileReader::GetRecordBatchReader(const std::vector&, const std::vector&, std::unique_ptr) virtual ::arrow::Status GetRecordBatchReader( ^~~~~~~~ /usr/include/parquet/arrow/reader.h:190:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadTable(std::shared_ptr) virtual ::arrow::Status ReadTable(std::shared_ptr<::arrow::Table> out) = 0; ^~~~~ /usr/include/parquet/arrow/reader.h:195:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadTable(const std::vector&, std::shared_ptr) virtual ::arrow::Status ReadTable(const std::vector& column_indices, ^~~~~ /usr/include/parquet/arrow/reader.h:198:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadRowGroup(int, const std::vector&, std::shared_ptr) virtual ::arrow::Status ReadRowGroup(int i, const std::vector& column_indices, ^~~~ /usr/include/parquet/arrow/reader.h:201:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadRowGroup(int, std::shared_ptr) virtual ::arrow::Status ReadRowGroup(int i, std::shared_ptr<::arrow::Table> out) = 0; ^~~~ /usr/include/parquet/arrow/reader.h:203:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadRowGroups(const std::vector&, const std::vector&, std::shared_ptr) virtual ::arrow::Status ReadRowGroups(const std::vector& row_groups, ^~~~~ /usr/include/parquet/arrow/reader.h:207:27: note: virtual arrow::Status parquet::arrow::FileReader::ReadRowGroups(const std::vector&, std::shared_ptr) virtual ::arrow::Status ReadRowGroups(const std::vector& row_groups, ^~~~~ /usr/include/parquet/arrow/reader.h:211:27: note: virtual arrow::Status parquet::arrow::FileReader::ScanContents(std::vector, int32_t, int64_t) virtual ::arrow::Status ScanContents(std::vector columns, ^~~~ /usr/include/parquet/arrow/reader.h:217:43: note: virtual std::shared_ptr parquet::arrow::FileReader::RowGroup(int) virtual std::shared_ptr RowGroup(int row_group_index) = 0; ^~~~ /usr/include/parquet/arrow/reader.h:220:15: note: virtual int parquet::arrow::FileReader::num_row_groups() const virtual int num_row_groups() const = 0; ^~~~~~ /usr/include/parquet/arrow/reader.h:222:30: note: virtual parquet::ParquetFileReader parquet::arrow::FileReader::parquet_reader() const virtual ParquetFileReader parquet_reader() const = 0; ^~~~~~ /usr/include/parquet/arrow/reader.h:226:16: note: virtual void parquet::arrow::FileReader::set_use_threads(bool) virtual void set_use_threads(bool use_threads) = 0; ^~~~~~~ parquet_impl.cpp: In function ‘ParquetFdwExecutionState create_parquet_state(const char, TupleDesc, MemoryContext, std::set&, bool, bool)’: parquet_impl.cpp:213:68: error: no matching function for call to ‘FromParquetSchema(const parquet::SchemaDescriptor&, std::shared_ptr)’ if (!parquet::arrow::FromParquetSchema(schema, &festate->schema).ok()) ^ In file included from parquet_impl.cpp:11:0: /usr/include/parquet/arrow/reader.h:328:17: note: candidate: arrow::Status parquet::arrow::FromParquetSchema(const parquet::SchemaDescriptor, const parquet::ArrowReaderProperties&, const std::shared_ptr&, std::shared_ptr) ::arrow::Status FromParquetSchema( ^~~~~ /usr/include/parquet/arrow/reader.h:328:17: note: candidate expects 4 arguments, 2 provided /usr/include/parquet/arrow/reader.h:334:17: note: candidate: arrow::Status parquet::arrow::FromParquetSchema(const parquet::SchemaDescriptor, const parquet::ArrowReaderProperties&, std::shared_ptr) ::arrow::Status FromParquetSchema(const SchemaDescriptor parquet_schema, ^~~~~ /usr/include/parquet/arrow/reader.h:334:17: note: candidate expects 3 arguments, 2 provided parquet_impl.cpp: At global scope: parquet_impl.cpp:502:35: error: ‘RowGroupStatistics’ is not a member of ‘parquet’ row_group_matches_filter(parquet::RowGroupStatistics stats, ^~~~~~ parquet_impl.cpp:502:35: note: suggested alternative: ‘BoolStatistics’ row_group_matches_filter(parquet::RowGroupStatistics stats, ^~~~~~ BoolStatistics parquet_impl.cpp:502:55: error: ‘stats’ was not declared in this scope row_group_matches_filter(parquet::RowGroupStatistics stats, ^~~~~ parquet_impl.cpp:502:55: note: suggested alternative: ‘stat’ row_group_matches_filter(parquet::RowGroupStatistics stats, ^~~~~ stat parquet_impl.cpp:503:42: error: expected primary-expression before ‘’ token arrow::DataType arrow_type, ^ parquet_impl.cpp:503:43: error: ‘arrow_type’ was not declared in this scope arrow::DataType arrow_type, ^~~~~~ parquet_impl.cpp:503:43: note: suggested alternative: ‘errdatatype’ arrow::DataType arrow_type, ^~~~~~ errdatatype parquet_impl.cpp:504:41: error: expected primary-expression before ‘’ token RowGroupFilter filter) ^ parquet_impl.cpp:504:42: error: ‘filter’ was not declared in this scope RowGroupFilter filter) ^~ parquet_impl.cpp:504:42: note: suggested alternative: ‘biteq’ RowGroupFilter filter) ^~ biteq parquet_impl.cpp:504:48: error: expression list treated as compound expression in initializer [-fpermissive] RowGroupFilter filter) ^ parquet_impl.cpp: In function ‘void extract_rowgroups_list(PlannerInfo, RelOptInfo)’: parquet_impl.cpp:615:64: error: invalid new-expression of abstract class type ‘parquet::arrow::FileReader’ false))); ^ parquet_impl.cpp:649:46: error: ‘RowGroupStatistics’ is not a member of ‘parquet’ std::shared_ptr stats; ^~~~~~ parquet_impl.cpp:649:46: note: suggested alternative: ‘BoolStatistics’ std::shared_ptr stats; ^~~~~~ BoolStatistics parquet_impl.cpp:649:46: error: ‘RowGroupStatistics’ is not a member of ‘parquet’ parquet_impl.cpp:649:46: note: suggested alternative: ‘BoolStatistics’ std::shared_ptr stats; ^~~~~~ BoolStatistics parquet_impl.cpp:649:64: error: template argument 1 is invalid std::shared_ptr stats; ^ parquet_impl.cpp:654:48: error: cannot convert ‘std::shared_ptr’ to ‘int’ in assignment stats = column->statistics(); ^ parquet_impl.cpp:657:37: error: ‘NodeToField’ is not a member of ‘parquet::arrow’ parquet::arrow::NodeToField(schema_node.field(k), &field); ^~~ parquet_impl.cpp:665:57: error: request for member ‘get’ in ‘stats’, which is of non-class type ‘int’ !row_group_matches_filter(stats.get(), type.get(), &filter)) ^~~ parquet_impl.cpp:665:83: error: ‘row_group_matches_filter’ cannot be used as a function !row_group_matches_filter(stats.get(), type.get(), &filter)) ^ In file included from /usr/include/postgresql/11/server/postgres.h:47:0, from parquet_impl.cpp:19: parquet_impl.cpp: In function ‘void initialize_castfuncs(ParquetFdwExecutionState, TupleDesc)’: parquet_impl.cpp:1011:64: error: ‘using element_type = class arrow::ChunkedArray {aka class arrow::ChunkedArray}’ has no member named ‘name’ festate->table->column(arrow_col)->name().c_str()), ^ /usr/include/postgresql/11/server/utils/elog.h:107:14: note: in definition of macro ‘ereport_domain’ errfinish rest; \ ^~~~ parquet_impl.cpp:1008:13: note: in expansion of macro ‘ereport’ ereport(ERROR, ^~~ parquet_impl.cpp: In function ‘void populate_slot(ParquetFdwExecutionState, TupleTableSlot, bool)’: parquet_impl.cpp:1332:50: error: ‘using element_type = class arrow::ChunkedArray {aka class arrow::ChunkedArray}’ has no member named ‘data’ if (++chunkInfo.chunk >= column->data()->num_chunks()) ^~~~ parquet_impl.cpp:1335:33: error: ‘using element_type = class arrow::ChunkedArray {aka class arrow::ChunkedArray}’ has no member named ‘data’ array = column->data()->chunk(chunkInfo.chunk).get(); ^~~~ parquet_impl.cpp: In function ‘bool read_next_rowgroup(ParquetFdwExecutionState, TupleDesc)’: parquet_impl.cpp:1500:34: error: ‘RowGroupStatistics’ is not a member of ‘parquet’ std::shared_ptr stats; ^~~~~~ parquet_impl.cpp:1500:34: note: suggested alternative: ‘BoolStatistics’ std::shared_ptr stats; ^~~~~~ BoolStatistics parquet_impl.cpp:1500:34: error: ‘RowGroupStatistics’ is not a member of ‘parquet’ parquet_impl.cpp:1500:34: note: suggested alternative: ‘BoolStatistics’ std::shared_ptr stats; ^~~~~~ BoolStatistics parquet_impl.cpp:1500:52: error: template argument 1 is invalid std::shared_ptr stats; ^ parquet_impl.cpp:1508:26: error: cannot convert ‘std::shared_ptr’ to ‘int’ in assignment ->statistics(); ^ parquet_impl.cpp:1511:51: error: base operand of ‘->’ is not a pointer festate->has_nulls[arrow_col] = (stats->null_count() > 0); ^~ parquet_impl.cpp:1538:47: error: ‘using element_type = class arrow::ChunkedArray {aka class arrow::ChunkedArray}’ has no member named ‘data’ festate->chunks.push_back(column->data()->chunk(0).get()); ^~~~ parquet_impl.cpp: In function ‘std::cxx11::list<std::pair<std::cxx11::basic_string, unsigned int> > extract_parquet_fields(ImportForeignSchemaStmt, const char)’: parquet_impl.cpp:1854:70: error: invalid new-expression of abstract class type ‘parquet::arrow::FileReader’ parquet::ParquetFileReader::OpenFile(path, false))); ^ parquet_impl.cpp:1865:71: error: no matching function for call to ‘FromParquetSchema(const parquet::SchemaDescriptor, std::shared_ptr)’ if (!parquet::arrow::FromParquetSchema(meta->schema(), &schema).ok()) ^ In file included from parquet_impl.cpp:11:0: /usr/include/parquet/arrow/reader.h:328:17: note: candidate: arrow::Status parquet::arrow::FromParquetSchema(const parquet::SchemaDescriptor, const parquet::ArrowReaderProperties&, const std::shared_ptr&, std::shared_ptr) ::arrow::Status FromParquetSchema( ^~~~~ /usr/include/parquet/arrow/reader.h:328:17: note: candidate expects 4 arguments, 2 provided /usr/include/parquet/arrow/reader.h:334:17: note: candidate: arrow::Status parquet::arrow::FromParquetSchema(const parquet::SchemaDescriptor, const parquet::ArrowReaderProperties&, std::shared_ptr) ::arrow::Status FromParquetSchema(const SchemaDescriptor parquet_schema, ^~~~~ /usr/include/parquet/arrow/reader.h:334:17: note: candidate expects 3 arguments, 2 provided Makefile:22: recipe for target 'parquet.o' failed make: *** [parquet.o] Error 1

[zilder]

Hi @hpolonskaya,

Are you using arrow-0.15 to build the extension? I haven't try this version yet. It looks like they have refactored FileReader, row statistics, probably something else. I'll need some time to make corresponding changes in parquet_fdw. It should work with version 0.14 though.

[hpolonskaya]

Yes it's arrow-0.15 version any ETA when it will be available on this version.

[zilder]

Already started working on this. Hopefully by the end of the week but uncertain.

[hpolonskaya]

thank you really appreciated

[zilder]

Hi @hpolonskaya,

There is a new branch pg12 with changes for arrow-0.15 (and postgres 12 as well). Can you please check if it works for you?

[hpolonskaya]

We have pg 11 , I can compile it and created extension but i got this ERROR: parquet_fdw: Operation not permitted

[zilder]

It looks like the file you are reading is not accessible by postgres user. Try changing permissions.

[hpolonskaya]

Working now thank you