search

typestreamio / typestream

Open Source streaming platform. Write and run typed data pipelines with a minimal, familiar syntax.
https://typestream.io
Apache License 2.0
64 stars 5 forks source link

tee command #78

Open dezren39 opened 8 months ago

dezren39 commented 8 months ago

outputs to a local file or another stream. could be used to output file data from repl while viewing might be nice to have a utcNow() function or tee -rolling 5m and then could accept other units of time. then one could output a new file every execution. dunno, maybe it's out of scope.

lucapette commented 8 months ago

dunno, maybe it's out of scope.

I don't think it is. In fact, the core suggestion that there should be a way to make TypeStream "mount" a local FileSystem into its operations has come up quite consistently in the conversations I had had.

I'm very curious to understand better what you're proposing with utcNow() and tee --rolling 5m. Especially the function, I think I understand the latter:

cat books | tee --rolling 5m books.ndjson

would roll books.ndjson.<datetime> files every 5 min? That seems pretty interesting also in the context of "mounting" object storage into TypeStream (s3 comes up all the time)

Is that what you have in mind?

dezren39 commented 8 months ago

yes that is exactly what i had in mind! certain kinds of log or data streaming, s3 mounting would be perfect here too. I had only thought through as far as local disk. At my work, we looked at mountpoint-s3 for managing app log outputs that had a similar rolling feature.

lucapette commented 8 months ago

Exciting!

I have to think a little about how to implement tee (I'm sure my brain will do that even if I try not to prio this, it's just too interesting!) before moving forward with it so I say we keep the issue open to reflect the current state: want to do, not sure yet how/when.