python-streamz / streamz

Real-time stream processing for python
https://streamz.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
1.24k stars 148 forks source link

how can we just convert streamz.dataframe into a normal dataframe? #183

Open fs-rd-anish-dey opened 6 years ago

fs-rd-anish-dey commented 6 years ago

i have this sdf

from streamz import Stream

source = Stream.filenames('~/jupyter/streams/csv/*.csv', poll_interval=0.1)
source.start()
example = pd.DataFrame({'cnt': [], 'word': []})
sdf = source.map(pd.read_csv).to_dataframe(example=example)
sdf

I just want to visualize it normally like I can do for a normal pandaz dataframe. Any help for visualizing the above sdf(example below) into a bar graph will be immensely helpful

|cnt|word|
+---+----+
|  3|   9|
|  1|   1|
|  1|   5|
|  3|   7|
|  3|  10|
|  3|   6|
|  2|   4|
|  2|   8|
+---+----+
mrocklin commented 6 years ago

My sense is that this would make more sense after we added support for stop or end-of-stream signals. I'm not sure how to accomplish this cleanly before then.

On Tue, Jun 5, 2018 at 3:32 AM, fs-rd-anish-dey notifications@github.com wrote:

i have this sdf

from streamz import Stream

source = Stream.filenames('~/jupyter/streams/csv/*.csv', poll_interval=0.1) source.start() example = pd.DataFrame({'cnt': [], 'word': []}) sdf = source.map(pd.read_csv).to_dataframe(example=example) sdf

I just want to visualize it normally like I can do for a normal pandaz dataframe. Any help for visualizing the above sdf(example below) into a bar graph will be immensely helpful

|cnt|word| +---+----+ | 3| 9| | 1| 1| | 1| 5| | 3| 7| | 3| 10| | 3| 6| | 2| 4| | 2| 8| +---+----+

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mrocklin/streamz/issues/183, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszGFkZAVeJabEHYHNlIAECJ5JwduMks5t5jPxgaJpZM4UaSyY .

CJ-Wright commented 6 years ago

Could we have a crude end-of-stream via a collect?