Can't fetch http query set

[waynexia]

Originally posted by @alamb: https://github.com/waynexia/datafusion-wasm/issues/1

Thanks @waynexia this is amazing 🙏 I found it from https://github.com/apache/arrow-datafusion/discussions/9834

I figured I would report an error I encountered in case that would help

I tried to setup https://waynexia.github.io/datafusion-playground/ to read from the clickbench data

https://github.com/ClickHouse/ClickBench/

However, when I ran this query

create external table hits stored as parquet location 'https://datasets.clickhouse.com/hits_compatible/hits.parquet';

datafusion error: Object Store error: Generic Unexpected error: Unexpected (temporary) at stat, context: { url: https://datasets.clickhouse.com:443/hits_compatible/hits.parquet, called: http_util::Client::send, service: http, path: hits_compatible/hits.parquet } => send http request, source: error sending request: JsValue(TypeError: Failed to fetch
TypeError: Failed to fetch
    at Dy (https://waynexia.github.io/datafusion-playground/assets/index-gHAd8CE7.js:9:2251)
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[14334]:0x1883492
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[199]:0x5388ce
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[629]:0x95fbcf
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[7872]:0x15833d4
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[12086]:0x17c1ac2
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[11177]:0x176f631
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[10984]:0x175ba20
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[580]:0x91e713
    at https://waynexia.github.io/datafusion-playground/assets/datafusion_wasm_bg-X0TXhe2e.wasm:wasm-function[11483]:0x178bd1a)

[waynexia]

Thanks for your reporting! I use the same parquet file in develop testing XD.

This issue is likely a CORS problem. You should find a request is blocked in the browser's debugging panel. The root cause is the underlying storage implement (OpenDAL HTTP operator) can't add a CORS header. Related discussion is here.

It's now a bit tricky to use the HTTP/HTTPS files. The solution is like using S3 that setting the server to add the needed CORS header. But HTTPs server doesn't have an easy way to add it like S3.

I've tried two ways to work around it: (a) download the file and serve it from the local environment using a server that can add custom headers. (b) setup a HTTPS proxy and add the header in that proxy. Both ways request a TLS cert since requesting HTTP content from HTTPS website (that playground) is also forbidden.

[alamb]

Thanks @waynexia -- now that you say this it makes sense that CORs would prevent the webpage from making requests to arbitrary other hosts (I forget now that I mostly do backend development these days).

The most interesting usecase for sure would be to show the wasm playground working with some data served from the same server (aka to do "in browser analytics")

[jiangzhx]

Give this a try. DuckDB's S3 address is already set up to allow cross-origin resource sharing (CORS), it's work for me.

CREATE EXTERNAL TABLE lineitem STORED AS PARQUET LOCATION 'https://shell.duckdb.org/data/tpch/0_01/parquet/lineitem.parquet';

[waynexia]

Thanks for this! It's great 👍

I added it to playground's readme https://github.com/datafusion-contrib/datafusion-wasm-playground/commit/2b86c19a28dc802892a1e16fe50ba2285bdfffd9

[waynexia]

Cross reference https://github.com/apache/opendal/issues/4446, ticket tries to solve the CORS problem from our side.

[Xuanwo]

As mentioned in https://github.com/apache/opendal/issues/4446, we can add headers while building opendal operators.