sqlsmith: Generate multiple input formats: Protobuf, AVRO, JSON

kwannoel commented 1 year ago

Motivation

Tests source parsing and reading.
We have encountered various bugs from source parser recently, this will help us catch bugs early.

Basic Requirements

Able to test basic source read of various row formats.
Able to test basic source parse of various row formats.

(Thanks @waruto210 for clarifying with me.)

Full Requirements

Able to generate complex nested schema.
Able to configure nesting depth.
Able to generate mix of data types, including array.
Main use is to test for correctness, but perhaps it can be used for performance testing in the future?

Background

Originally posted here: https://github.com/risingwavelabs/risingwave/issues/5164 by @neverchanje :

Source reading/parsing

The parsing part will be more deterministic than the rest. We need to generate random data in a specific format (with a probability of generating false data, the expected behavior is to drop it) and verify the correctness of the parsed output.

[ ] MockSource x Protobuf

[ ] MockSource x JSON https://github.com/risingwavelabs/risingwave/pull/5512

[ ] MockSource x Debezium JSON

[ ] MockSource x Avro

Offline discussion with @neverchanje :

The goal is to stabilize of protobuf and avro, which typically have complex nested schema. For testing, we need to ensure that a protobuf file with multiple nested levels and a mix of various data types (including array) can be correctly parsed.

kwannoel commented 1 year ago

~Not sure if right idea... Re-open when have more details.~

kwannoel commented 1 year ago

cc @jon-chuang @neverchanje @tabVersion @waruto210

waruto210 commented 1 year ago

I propose to make datagen connector support generating multiple formats including native chunk, while nexmark only supports native.

kwannoel commented 1 year ago

I propose to make datagen connector support generating multiple formats including native chunk, while nexmark only supports native.

+1 for this.

kwannoel commented 1 year ago

FYI we can do this in next-release. Don't want to block https://github.com/risingwavelabs/risingwave/pull/7612.

fuyufjh commented 1 year ago

cc. @tabVersion

tabVersion commented 1 year ago

FYI we can do this in next-release. Don't want to block https://github.com/risingwavelabs/risingwave/pull/7612.

time to start?

neverchanje commented 1 year ago

Testing is always hard 😢

kwannoel commented 1 year ago

Have quite a bit of backlog at the moment. Don't plan to work on this anytime soon 🤢 maybe after this quarter.

risingwavelabs / risingwave