fatal error: 'inet_extension.hpp' file not found

yurivict commented 1 month ago

What happens?

The build fails to find the file "inet_extension.hpp":

In file included from /usr/ports/databases/duckdb/work/.build/src/main/extension/ub_duckdb_main_extension.cpp:3:
In file included from /usr/ports/databases/duckdb/work/duckdb-1.1.0/src/main/extension/extension_helper.cpp:57:
In file included from /usr/ports/databases/duckdb/work/duckdb-1.1.0/src/include/duckdb/main/extension/generated_extension_loader.hpp:17:
/usr/ports/databases/duckdb/work/.build/codegen/include/generated_extension_headers.hpp:8:10: fatal error: 'inet_extension.hpp' file not found
    8 | #include "inet_extension.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~
1 error generated.

This file is present:

$ find . -name "inet_extension.hpp"
./work/duckdb-1.1.0/extension/inet/src/include/inet_extension.hpp

Same happens for other extensions:

In file included from /usr/ports/databases/duckdb/work/.build/src/main/extension/ub_duckdb_main_extension.cpp:3:
In file included from /usr/ports/databases/duckdb/work/duckdb-1.1.0/src/main/extension/extension_helper.cpp:57:
In file included from /usr/ports/databases/duckdb/work/duckdb-1.1.0/src/include/duckdb/main/extension/generated_extension_loader.hpp:17:
/usr/ports/databases/duckdb/work/.build/codegen/include/generated_extension_headers.hpp:12:10: fatal error: 'sqlsmith_extension.hpp' file not found
   12 | #include "sqlsmith_extension.hpp"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Version: 1.1.0 clang-18 FreeBSD 14.1

To Reproduce

regular configure/build

cmake arguments: -DOVERRIDE_GIT_DESCRIBE=v1.1.0-0-g0000000000 -DBUILD_EXTENSIONS="autocomplete;fts;httpfs;icu;inet;jemalloc;json;parquet;sqlsmith;tpcds;tpch;sqlite_scanner" -DDUCKDB_MAJOR_VERSION=1 -DDUCKDB_MINOR_VERSION=1 -DDUCKDB_PATCH_VERSION=0 -DBUILD_TPCE:BOOL=true -DCMAKE_C_COMPILER:STRING="cc" -DCMAKE_CXX_COMPILER:STRING="c++" -DCMAKE_C_FLAGS:STRING="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " -DCMAKE_C_FLAGS_DEBUG:STRING="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " -DCMAKE_C_FLAGS_RELEASE:STRING="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing -DNDEBUG" -DCMAKE_CXX_FLAGS:STRING="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing -I/usr/ports/databases/duckdb/work/duckdb-1.1.0/extension/sqlite_scanner/src/include " -DCMAKE_CXX_FLAGS_DEBUG:STRING="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing -I/usr/ports/databases/duckdb/work/duckdb-1.1.0/extension/sqlite_scanner/src/include " -DCMAKE_CXX_FLAGS_RELEASE:STRING="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing -I/usr/ports/databases/duckdb/work/duckdb-1.1.0/extension/sqlite_scanner/src/include -DNDEBUG" -DCMAKE_EXE_LINKER_FLAGS:STRING=" -fstack-protector-strong " -DCMAKE_MODULE_LINKER_FLAGS:STRING=" -fstack-protector-strong " -DCMAKE_SHARED_LINKER_FLAGS:STRING=" -fstack-protector-strong " -DCMAKE_INSTALL_PREFIX:PATH="/usr/local" -DCMAKE_AUTOGEN_PARALLEL:STRING="7" -DCMAKE_BUILD_TYPE:STRING="Release" -DTHREADS_HAVE_PTHREAD_ARG:BOOL=YES -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=YES -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DFETCHCONTENT_FULLY_DISCONNECTED:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=ON -DBUILD_UNITTESTS:BOOL=OFF -GNinja

ninja is used as a backend.

OS:

FreeBSD 14.1

DuckDB Version:

1.1.0

DuckDB Client:

n/a

Hardware:

No response

Full Name:

Yuri Victorovich

Affiliation:

n/a

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have not tested with any build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

[X] Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

[X] Yes, I have

carlopi commented 1 month ago

Here the problem is that inet and sqlsmith extensions have been moved from being just folders in the duckdb/duckdb repository to being external independent repositories at https://github.com/duckdb/duckdb_inet and https://github.com/duckdb/duckdb_sqlsmith.

You would need to replicate a similar logic to the one you are using for sqlite_scanner, that is also an external repository.

Note that a automatic sort of option is passing to cmake: -DCORE_EXTENSIONS="autocomplete;fts;httpfs;icu;inet;jemalloc;json;parquet;sqlsmith;tpcds;tpch" instead of `-DBUILD_EXTENSION="a;b;c", but that will rely on fetching directly from github, that I guess you are trying to avoid as part of the build step.

Note that other extensions will in the future move from being just folders in the repo to external repositories, since that help us decouple logic / changes, and allow more granular pick-and-choose of features.

Proper solution to this issue would be documenting this process in the docs and possibly make so the error message is more friendly, but I though of giving some context hoping that would help unblock you.

yurivict commented 1 month ago

Can DCORE_EXTENSIONS and DBUILD_EXTENSION lists be merged and configure do the right thing based on the flags assigned to the extension in its definition?

It doesn't seem like the user needs to know the difference between a core extension and external repository extension. Once they are present in the right location the build should just pick them up the same way.

carlopi commented 1 month ago

This does make sense, and it's the logic behind introducing the CORE_EXTENSIONS, that is mostly a stand-in replacement for BUILD_EXTENSIONS.

What's meant to work, and it's tested, is that:

CORE_EXTENSIONS="a;b;c;d" GEN=ninja make

or equivalent:

cmake -DCORE_EXTENSIONS="a;b;c;d" .....

will fetch, build and link in the duckdb libraries extensions a, b, c and d, either from the duckdb repo or from the commits specified at .github/config/out_of_core.cmake.

You can probably provide your own config files, put that instead of out_of_core.cmake, and it will the likely work.

duckdb / duckdb