search

lukehutch / tiny-reactive-dataflow

A tiny reactive dataflow library for scheduling a DAG of async functions in Javascript
MIT License
8 stars 0 forks source link

Question: How to adapt to a computed value database #1

Open lveillard opened 1 year ago

lveillard commented 1 year ago

Hello! Great package, doing some tests, and it looks great and performant!

I was wondering how would you approach the use case I'm working with. We have some computed values with lots of dependencies (multiple levels of dependencies) that are computed in the frontend and the backend.

The front-end is easy, when the user changes some value, we update our store which also runs a dataflow.set() with the new values. Getting the right values in the store.

The backend is more tricky. Imagine that you have a database with an interface, for instance a lambda, that gathers requests and tries to achieve the same result:

client [api]<> interface (engine) [lambda] <> database [postgres]

We are currently doing it like this in the lambda: 1) Check the impacts (reverse deps) of the input variables 2) get the required values in DB 3) Do all the computing topologically 4) send all the results to the DB

Receiving concurrent mutations without shared dependencies / impacts are fine. But mutations that have shared dependencies/impacted values do generate issues.

How would you approach this use case where lambdas can asynchronously change values of this reactive database?

📝One workaround is to do only onRead computed values and not onWrite computes, but we need both for performance reaons

lukehutch commented 1 year ago

Hey, congrats on filing bug #1 :) I'm glad you found this library! It has proved to be a killer technology for a complex data-driven UI I was working on.

Ok, so your problem is that your database reads and writes, for a single dataflow propagation, need to be wrapped inside a transaction. That way if the output values of one concurrent dataflow pass change the input values of a second concurrent dataflow pass after they have already been read, the second dataflow will be reverted (then you just retry the second transaction, it should succeed the next time).

lveillard commented 1 year ago

Happy to be the first one! I was looking for complex solutions with timestamps lol. Thanks, that will fix it!

By the way if we end up using this one instead of topologica (which seems it will be the case) would you accept a PR that gets rid of lodash? :)

Also, would it make sense for you to deploy it to npm?

lukehutch commented 1 year ago

Yes, certainly, however the lodash equals-testing method would be very hard to replicate in your own code, because proper equals-testing in Javascript is insanely complex.

I'm happy to deploy to npm too, I just have no experience with that.

I'm actually considering rewriting this in Dart, and it could be compiled to Javascript from there. I have no idea what the gotchas will be of this approach, but Dart is a far superior language to JS, and all my work is in Dart these days.