vinum-team / Vinum

A modern reactive state management library for correctness and speed.
MIT License
16 stars 1 forks source link

Add Data Streaming implemented using Observable Pattern #30

Closed sinlerdev closed 6 months ago

sinlerdev commented 1 year ago

More on this later.

sinlerdev commented 8 months ago

Sometimes it's better for the programmer to set up "pure" streams of data that flow through chains of state structs/objects (The struct is a better-suited name, but that's not the point of this post). Or in other words, it's often needed to have a utility in Vinum that makes it easy to send data through a graph without data loss (data loss in this case is preserving past data in order in the case of having no dependents).

And that's where Observables make sense (at least in my mind and according to my understanding of the pattern), as they are objects that initiate a data stream and then close it when its done. For example, you could have a WorkspaceChildrenStream which could be an observable that sends/fires the children of Workspace and then the observable could be killed whenever needed.


A possible API design for this, is to introduce a Stream function that you fire it with a "control" function that runs once the stream struct gets dependents. Something like this:


-- This stream will wait for 3 seconds before sending data
local myStream = Stream(function(send, complete)
   task.wait(3)
   for i = 1, 10 do
      send(i)
   end
   complete()
end)

On(myStream, function(data)
   print(data)
end)

-- output 
-- 1
-- 2
-- ...
-- 10

Though an important question is how will the rest of the library work with this feature? Should we implement this as a state struct with the special streaming behavior (and have the benefit of auto integration with the rest of the library), or define a new type that manages streams/streamers?

Moreover, how will Operators::Read behave with this one? Should it get the array of the entire sent data, or just the latest (the latter makes more sense than the former, as normally you would want to get the latest in a Compute for example)? The semantics of Reading is important, as all other important features (such as Operators::Use) depend either on the standard read function, or its semantics and behavior.

sinlerdev commented 8 months ago

Another important detail to note is the behavior of data preservation. Shall we preserve all data that is being sent and then resend it to all newly-connected dependents? Or just run the streaming function when the first dependent is added and not care about new dependents?

sinlerdev commented 8 months ago

Should we strictly stick to Observable's pattern or design something that is more similar to what people think of a stream? Upon further research, I think that sticking to the observable's pattern makes sense more as it allows for greater flexibility and removes the need for complicated data preservation behavior/implementation, however, it does still need some changes to the internal node infrastructure to allow controlled data forwarding to dependents.

sinlerdev commented 7 months ago

It might be more sensible to instead support Channels as independent state structs rather than internal ones. This being said, Streams would be nothing but basic containers of producer functions:

-- a Stream is just now {type = "stream", fn: yourFn}
local numberStream = Stream(function(channel: state)
    for i = 1, 10 do 
        Write(channel, i)
   end
end)
-- A Channel is a state struct that has some special behavior.
-- the most notable is that it starts updating the moment the first dependent
-- is connected.
local myChannel = Channel(numberStream)

-- Now that On is connected to myChannel, my channel will start running the producer function
-- previously defined by Stream, and send the data to our On connection.
On(myChannel, function() 
    print(Read(myChannel)
end)
xiyler commented 6 months ago

Upon further inspection, this overcomplicates the codebase and the internal graph system that supports Vinum. It's recommended to use a separate Observable library with Vinum.