Populate user guide page for streaming

pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust

https://docs.pola.rs

Other

30.23k stars 1.95k forks source link

Populate user guide page for streaming #13680

Open braaannigan opened 10 months ago

braaannigan commented 10 months ago

Description

The current page on streaming is under construction. Some points of interest:

what is streaming
how do we use streaming
when streaming works
chunk size and threading
streaming from cloud storage

Let me know if there's anything else. I've assigned this to myself

Link

https://docs.pola.rs/user-guide/lazy/streaming/

braaannigan commented 10 months ago

I've remembered there is a basic streaming page in the concepts section.

Any preferences whether we want to develop that one or the page in the lazy API section?

deanm0000 commented 10 months ago

One thing that has caused confusion is the distinction between lazy and streaming.

It would be good to explicitly spell out that laziness is necessary for streaming but that not all lazy is automatically streaming. The streaming engine works differently so many(most) operations will be slower but some might be faster.