search

quackscience / duckdb-extension-httpclient

DuckDB HTTP GET/POST Client in a Community Extension
https://duckdb.org/community_extensions/extensions/http_client.html
MIT License
40 stars 3 forks source link

Add new http_get extra function with header & params #8

Closed ahuarte47 closed 1 week ago

ahuarte47 commented 2 weeks ago

This PR implements https://github.com/quackscience/duckdb-extension-httpclient/issues/7

lmangani commented 2 weeks ago

Thanks @ahuarte47! I'm trying to understand why the MINGW buileder fails as we seem to correctly link everything in CMake....

lmangani commented 2 weeks ago

since the http_post function seems fine I guess the issue is in the differences between the two not using linked httplib functions somehow. I can take a look later tonight.

lmangani commented 2 weeks ago

I couldn't find a way to make MGW happy from the linker without code changes. Since the only difference appear to be the params, a method to avoid this could be to build the URL with parameters manually before making the request, but I did not try.

An easier option - How about excluding windows_mgw from builds @ahuarte47 ?

ahuarte47 commented 2 weeks ago

I do not see any problem building the URL with parameters manually. I can change code this weekend, or if you prefer you can modify yourself.

ahuarte47 commented 1 week ago

Hi @lmangani I do not understand why MGW can not build the extension now. Please, can you share the error message? httplib is expanding the path with parameters as well, so, your idea is great, but the logic in httplib seems more elaborate.

Maybe is the new template function to convert input parameters the guilty?

lmangani commented 1 week ago

Hola Muy Buenas @ahuarte47 I also do not understand where all the unresolved symbols come from. I wonder if it has to to with the latest version of DuckDB since our code didn't really change much.

libcrypto.a(libcrypto-lib-dso_win32.obj) : error LNK2001: unresolved external symbol sprintf [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-rsaz_exp.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-e_capi.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cmp_util.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bn_exp.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-curve448.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cryptlib.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-getenv.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-err_prn.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-o_fopen.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libssl.a(libssl-lib-tls_common.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libssl.a(libssl-lib-extensions_srvr.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-o_dir.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-srp_vfy.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-v3_utl.obj) : error LNK2019: unresolved external symbol sscanf referenced in function ipv4_from_asc [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-http_lib.obj) : error LNK2001: unresolved external symbol sscanf [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol WspiapiGetNameInfo referenced in function addr_strings.part.1 [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol gai_strerrorA referenced in function addr_strings.part.1 [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol WspiapiFreeAddrInfo referenced in function BIO_ADDRINFO_free [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol WspiapiGetAddrInfo referenced in function BIO_lookup [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cryptlib.obj) : error LNK2019: unresolved external symbol __imp__vsnwprintf referenced in function OPENSSL_showfatal [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cryptlib.obj) : error LNK2019: unresolved external symbol __imp__vsnprintf referenced in function OPENSSL_showfatal [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-a_time.obj) : error LNK2019: unresolved external symbol __imp__timezone referenced in function ossl_asn1_string_to_time_t [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet.duckdb_extension : fatal error LNK1120: 10 unresolved externals [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
ahuarte47 commented 1 week ago

Hola Muy Buenas @ahuarte47 I also do not understand where all the unresolved symbols come from. I wonder if it has to to with the latest version of DuckDB since our code didn't really change much.

libcrypto.a(libcrypto-lib-dso_win32.obj) : error LNK2001: unresolved external symbol sprintf [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-rsaz_exp.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-e_capi.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cmp_util.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bn_exp.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-curve448.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cryptlib.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-getenv.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-err_prn.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-o_fopen.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libssl.a(libssl-lib-tls_common.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libssl.a(libssl-lib-extensions_srvr.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-o_dir.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-srp_vfy.obj) : error LNK2001: unresolved external symbol ___chkstk_ms [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-v3_utl.obj) : error LNK2019: unresolved external symbol sscanf referenced in function ipv4_from_asc [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-http_lib.obj) : error LNK2001: unresolved external symbol sscanf [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol WspiapiGetNameInfo referenced in function addr_strings.part.1 [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol gai_strerrorA referenced in function addr_strings.part.1 [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol WspiapiFreeAddrInfo referenced in function BIO_ADDRINFO_free [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-bio_addr.obj) : error LNK2019: unresolved external symbol WspiapiGetAddrInfo referenced in function BIO_lookup [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cryptlib.obj) : error LNK2019: unresolved external symbol __imp__vsnwprintf referenced in function OPENSSL_showfatal [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-cryptlib.obj) : error LNK2019: unresolved external symbol __imp__vsnprintf referenced in function OPENSSL_showfatal [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
libcrypto.a(libcrypto-lib-a_time.obj) : error LNK2019: unresolved external symbol __imp__timezone referenced in function ossl_asn1_string_to_time_t [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]
D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet.duckdb_extension : fatal error LNK1120: 10 unresolved externals [D:\a\duckdb-extension-httpclient\duckdb-extension-httpclient\build\release\extension\parquet\parquet_loadable_extension.vcxproj]

Hi Lorenzo, can you run make clean + make test to refresh the complete build?

lmangani commented 1 week ago

Hey @ahuarte47

Hi Lorenzo, can you run make clean + make test to refresh the complete build?

The errors are from the Github action builder, not from my local setup. (I don't have a Windows MINIGW env to test with)

lmangani commented 1 week ago

@ahuarte47 i excluded the windows_amd64_mingw for the time being. We can figure this out later but since 99.9% of the userbase for this extension is on Linux and OSX releasing your new features seems more important. If you agree just pull main back into the branch and it should pass. I'll take care of doing the same on the extensions repository. Vale?

ahuarte47 commented 1 week ago

Ah, thanks, sorry for my misunderstanding. Ok I see your commit https://github.com/quackscience/duckdb-extension-httpclient/commit/640fc62d96286bf4ff73fafd576431fa95d23cd7

ahuarte47 commented 1 week ago

@ahuarte47 i excluded the windows_amd64_mingw for the time being. We can figure this out later but since 99.9% of the userbase for this extension is on Linux and OSX releasing your new features seems more important. If you agree just pull main back into the branch and it should pass. I'll take care of doing the same on the extensions repository. Vale?

Yeah!!! I totaly agree! Thanks @lmangani