search

neilramaswamy / structured-streaming

Guides, documentation, and an API reference for Apache Spark Structured Streaming.
https://structured-streaming.vercel.app
Apache License 2.0
0 stars 0 forks source link

Source examples should be complete #24

Open neilramaswamy opened 7 months ago

neilramaswamy commented 7 months ago

Previously, we've held the philosophy that examples for specific features (like sources) should be minimal.

However, it's often useful to be able to see the schema of certain development sources (like rate) being used, because the schema that they have is non-obvious. We don't need to actually call .writeStream to a real sink in these examples, but we should at least do some transformation.

neilramaswamy commented 3 months ago

Also source examples should mention that not all formats support configs like admission control (from some tests I've run, it seems like json doesn't behave the same way as delta w.r.t. admission control)