search

krish-adi / barfi

Python Flow Based Programming environment that provides a graphical programming environment.
https://barfi.readthedocs.io
MIT License
659 stars 71 forks source link

a non backend coupled version, like an open API version #11

Open osi1880vr opened 2 years ago

osi1880vr commented 2 years ago

Hey krish-adi,

would it be thinkable to have a decoupled version of barfi with no strings attached to streamlit or nodejs? I understand there is the backend which need to be present in some way, how you would document the backend API and one could setup his own backend? Once the build backend is working it could be shared too, maybe that way more implementations show up.

We try to use your or any similar system but they all come bound to some kind of a server which I dont need or dont like. I would like to have a totaly open system where I could implement my own backend. Yes thats additional work on my side but at least I would have more control about how and where I implement the system. Like there is a way to have a browser only barfi but the backend has to be build by providing APIs which provide what ever barfi needs in the backend to work.

Sounds weired but might be an idea after all ;)

Thanks for your time and also thanks for all the effort you spend building barfi, we already have plenty of fun with it even with the "limitations" of being nailed to use streamlit for now.

Cheers Werner

krish-adi commented 2 years ago

Hey @osi1880vr ,

  1. A decoupled version, by implementing your own backend, would that include the GUI component as well? If so, would that also mean that you would want to have the ability to add the GUI component to any frontend?
  2. Without the GUI, the, barfi is decoupled from streamlit already. All the you need is a way to create schemas. If you have a schema of the blocks and their connections, you can just pass it to the compute_engine. If this is what you are looking for I can update the documentation.
krish-adi commented 2 years ago

Also, I am thinking of a way to decouple the GUI and the backend. As in, to have the GUI independent and be able to add it to any interface.

If you have any suggestions on how to achieve this, I'd welcome it very much :)

ldmtwo commented 4 months ago

I would like to collaborate on this. The way I would approach this is to create a version based on Flask or FastAPI as an alternate starting point for devs. This would help identify what is core to the web UI and which parts are Streamlit interfaces. The other core parts are loading the schemas.pkl and executing the DAG.

This tool is incredible to have mixed with python, but I'm not sure how to extend it and make it easy for a to use. I think a few tweaks would get it there. Block and schema management and navigation are the current pain point I've had. I could easily create 100 blocks and 100 design schemas, but I don't want to see 100 in the menu and I don't yet know how to swap schemas.barfi. We need a way to manage the menu better. I'm just rattling out ideas, but I've also been thinking of how to swap out the execution part with Airflow. I like Barfi for experimental design and management, but for a scaled up execution of the graph, I would want to use either Pyspark or Airflow to get all the benefits of those. To get the needed parallelism and laziness in Barfi, I would have to create a generator of futures for each output.

If you add more docs about the tools you used, that would help me at least dive deeper into the frontend integration.