Unable to compile pg_duckdb repo

[BGireesh]

OS Details

Postgres version details

Error Logs:

root@6e2d4f727e78:/pg_duckdb# make
rm -f -f .depend
g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/scan/heap_reader.o src/scan/heap_reader.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/scan/index_scan_utils.o src/scan/index_scan_utils.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/scan/postgres_index_scan.o src/scan/postgres_index_scan.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/scan/postgres_scan.o src/scan/postgres_scan.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/scan/postgres_seq_scan.o src/scan/postgres_seq_scan.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/utility/copy.o src/utility/copy.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/vendor/pg_explain.o src/vendor/pg_explain.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_metadata_cache.o src/pgduckdb_metadata_cache.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_detoast.o src/pgduckdb_detoast.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_duckdb.o src/pgduckdb_duckdb.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_filter.o src/pgduckdb_filter.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_hooks.o src/pgduckdb_hooks.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_memory_allocator.o src/pgduckdb_memory_allocator.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_node.o src/pgduckdb_node.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_options.o src/pgduckdb_options.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_planner.o src/pgduckdb_planner.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb_types.o src/pgduckdb_types.cpp >> .depend; g++ -Iinclude -Ithird_party/duckdb/src/include -Ithird_party/duckdb/third_party/re2 -std=c++17 -Wno-sign-compare  -I. -I./ -I/usr/include/postgresql/16/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=3 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/postgresql -I/usr/include/postgresql/16/server -MM -MT src/pgduckdb.o src/pgduckdb.cpp >> .depend;
src/scan/heap_reader.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from src/scan/index_scan_utils.cpp:1:
include/pgduckdb/scan/index_scan_utils.hpp:4:10: fatal error: postgres.h: No such file or directory
    4 | #include "postgres.h"
      |          ^~~~~~~~~~~~
compilation terminated.
src/scan/postgres_index_scan.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/scan/postgres_scan.cpp:1:10: fatal error: duckdb/main/client_context.hpp: No such file or directory
    1 | #include "duckdb/main/client_context.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
src/scan/postgres_seq_scan.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/utility/copy.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/vendor/pg_explain.cpp:12:10: fatal error: postgres.h: No such file or directory
   12 | #include "postgres.h"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_metadata_cache.cpp:2:10: fatal error: postgres.h: No such file or directory
    2 | #include "postgres.h"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_detoast.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_duckdb.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_filter.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_hooks.cpp:2:10: fatal error: postgres.h: No such file or directory
    2 | #include "postgres.h"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_memory_allocator.cpp:1:10: fatal error: duckdb/common/allocator.hpp: No such file or directory
    1 | #include "duckdb/common/allocator.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
src/pgduckdb_node.cpp:2:10: fatal error: duckdb.hpp: No such file or directory
    2 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_options.cpp:2:10: fatal error: duckdb.hpp: No such file or directory
    2 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_planner.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb_types.cpp:1:10: fatal error: duckdb.hpp: No such file or directory
    1 | #include "duckdb.hpp"
      |          ^~~~~~~~~~~~
compilation terminated.
src/pgduckdb.cpp:2:10: fatal error: postgres.h: No such file or directory
    2 | #include "postgres.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:102: .depend] Error 1
root@6e2d4f727e78:/pg_duckdb#

[wuputah]

Try git submodule update --init --recursive and also make sure you have all the prerequisite packages: https://github.com/duckdb/pg_duckdb/blob/main/.github/workflows/build_and_test.yaml#L62-L64

[LeandroFernandes5]

I've installed the prerequisite packages but do face the same issue reported above.

[wuputah]

fatal error: duckdb.hpp: No such file or directory

This is fixed by git submodule update --init --recursive. There was an attempt to automatically run this in the Makefile, but it seems to not be working.

fatal error: postgres.h: No such file or directory

This means you do not have the Postgres dev / header package installed, apt install postgresql-server-dev-16 (or 17).

[BGireesh]

fatal error: duckdb.hpp: No such file or directory

This is fixed by git submodule update --init --recursive. There was an attempt to automatically run this in the Makefile, but it seems to not be working.

fatal error: postgres.h: No such file or directory

This means you do not have the Postgres dev / header package installed, apt install postgresql-server-dev-16 (or 17).

It kinda worked! But ran into another issue

ckdb_detoast.o src/pgduckdb_detoast.cpp
src/pgduckdb_detoast.cpp:9:10: fatal error: lz4.h: No such file or directory
    9 | #include <lz4.h>
      |          ^~~~~~~
compilation terminated.
make: *** [<builtin>: src/pgduckdb_detoast.o] Error 1
# 

[BGireesh]

fatal error: duckdb.hpp: No such file or directory

This is fixed by git submodule update --init --recursive. There was an attempt to automatically run this in the Makefile, but it seems to not be working.

fatal error: postgres.h: No such file or directory

This means you do not have the Postgres dev / header package installed, apt install postgresql-server-dev-16 (or 17).

It kinda worked! But ran into another issue

ckdb_detoast.o src/pgduckdb_detoast.cpp
src/pgduckdb_detoast.cpp:9:10: fatal error: lz4.h: No such file or directory
    9 | #include <lz4.h>
      |          ^~~~~~~
compilation terminated.
make: *** [<builtin>: src/pgduckdb_detoast.o] Error 1
# 

apt-get install liblz4-dev

☝️ does the magic

[isaiasanchez]

Nice to have this into the README.md

[LeandroFernandes5]

I do agree with @isaiasanchez , both of these packages don't seem to be either mentioned on the README.md nor on the build_and_test.yml (https://github.com/duckdb/pg_duckdb/blob/main/.github/workflows/build_and_test.yaml#L62-L64).

What do you think @wuputah ?

[wuputah]

I realize that building the project is the only option right now, but this is a temporary thing. Because build instructions such as required packages are OS-, distro-, and distro-release-specific, I'm not planning on maintaining exact build instructions. Some of that will come out of building and releasing binaries, at which point most users should not need to compile. At that point, the expectation that building will principally be for developers, and that you can debug these types of issues on your own (or with the assist of Google, etc).

We could certainly add liblz4-dev to the list of packages that the CI installs, though I'm not sure why CI succeeds without it; most likely it's already installed in Github's base image due to some other piece of software that Github installs by default.

The Postgres dev package is covered under the precondition that you have the "standard set of build tools for building Postgres extensions." This is not needed in CI because Postgres is being compiled from source. The specific package you will need will depend on the version of Postgres you are using and how it is packaged.

What would be a good PR is fixing this issue is this, which is specific to this project:

There was an attempt to automatically run this in the Makefile, but it seems to not be working.

This appears to be a "dependency chain" issue, where the main compilation tries to start running before the code in the submodule is finished being checked out.