ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.81k stars 220 forks source link

Invalid unstructured json schema in source is not detected at plan time #729

Closed mwylde closed 2 months ago

mwylde commented 2 months ago

This query DDL is invalid because it specifies an unstructured JSON schema, but it does not have a single column of type TEXT/JSON:

CREATE TABLE source (
  x INT
) WITH (
    connector = 'websocket',
    endpoint = 'wss://ws-feed.exchange.coinbase.com',
    subscription_message = '{
      "type": "subscribe",
      "product_ids": [
        "BTC-USD"
      ],
      "channels": ["ticker"]
    }',
    format = 'json',
    'json.unstructured' = 'true'
);

This should produce a plan-time failure, but it does not. Instead it produces a runtime failure:

2024-08-30T18:08:30.232087Z ERROR arroyo_server_common: panicked at crates/arroyo-formats/src/de.rs:319:14:
no 'value' column for RawString format panic.file="crates/arroyo-formats/src/de.rs" panic.line=319 panic.column=14