search

dashbitco / broadway_dashboard

Keep track of your Broadway pipelines from Phoenix LiveDashboard
https://elixir-broadway.org/
Apache License 2.0
213 stars 17 forks source link

Add first working version of the page #1

Closed philss closed 3 years ago

philss commented 3 years ago

This ~commit~ PR introduces the first version of Broadway dashboard as an additional page for the Phoenix LiveDashboard.

It works by recording events from the pipelines and displaying a graph with each stage as a circle. The circles turn red when they are more busy and green and they are less busy.

This work is based on Marlus Saraiva's code from his presentation to ElixirConf 2019. In that talk he presented a graph showing the work of a Broadway pipeline, which is essentially the same from this change. See: https://www.youtube.com/watch?v=tPu-P97-cbE

Demo

https://user-images.githubusercontent.com/381213/119524701-40fa9a80-bd54-11eb-8dba-9da4d85c7a82.mp4

TODO

josevalim commented 3 years ago

This looks great! There is only one issue which is about the use of :rpc.call. When you do :rpc.call, the module you are calling on the remote node needs to exist on the remote node. LiveDashboard addresses this by loading a SystemInfo module in the remote node.

In our case, we don't do so, so it is likely that this will fail. You can probably reproduce this by having the Broadway topology run in a node and the dashboard in another. Then you call Node.connect so they can see each other and switching it on the menu.

josevalim commented 3 years ago

And the above is going to be complicated because you need to include all of the code that :rpc.call calls, so the counters, telemetry, etc.

philss commented 3 years ago

This looks great! There is only one issue which is about the use of :rpc.call. When you do :rpc.call, the module you are calling on the remote node needs to exist on the remote node. LiveDashboard addresses this by loading a SystemInfo module in the remote node.

@josevalim Thanks for pointing out! I was able to do something similar, which is to loading all the modules necessary to track the events in the remote node.

I was struggling at finding out a way to start the Metrics server in the remote node, so I decided to start it with Node.spawn/3 and without a link. This way I can start the server "there" and terminate it when the Dashboard node disconnects. Do you think this is a fair solution?

josevalim commented 3 years ago

This way I can start the server "there" and terminate it when the Dashboard node disconnects. Do you think this is a fair solution?

Yeah, I think so.

philss commented 3 years ago

@josevalim I think there is enough tests for now! But let me know if you think something needs a special look.

Also I changed the PipelineGraphComponent to make it more generic before open the other PR for the PhoenixLiveDashboard.

PS: I made some usage of Mix.install/1 in some distribution tests so I could run heterogeneous nodes. Let me know if this is acceptable :see_no_evil:

philss commented 3 years ago

It's almost there :grimacing: :point_down: almost-there

philss commented 3 years ago

As we discussed, the improvements are going to be tackled in other PRs.