[Research] Better layout algorithm options for Hamilton UI dataflows

DAGWorks-Inc / hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

https://hamilton.dagworks.io/en/latest/

BSD 3-Clause Clear License

1.87k stars 124 forks source link

[Research] Better layout algorithm options for Hamilton UI dataflows #1037

Open skrawcz opened 4 months ago

skrawcz commented 4 months ago

Is your feature request related to a problem? Please describe. The hamilton UI layout algorithm could be improved. Things can get mixed / muddled.

Describe the solution you'd like Have a better layout algorithm or optimization toggle.

Describe alternatives you've considered N/A

Additional context Try the Hamilton UI with the Hamilton UI example, and then toggle the various options on layout, and you'll see an example like the one below. The hard part is that this DAG dependent.

Screen Shot 2024-07-18 at 2 36 12 PM

skrawcz commented 4 months ago

This is an open ended task as it requires a bit of research.

zilto commented 4 months ago

The entanglement is due to the DAG order conflicting with the cluster order. My initial solution for the Hamilton VSCode extension was to redirect edges from/to nodes inside a cluster to the cluster. This also allows for "bundling" (tried to draw this idea).

Then, the user can view the granular edges by clicking on the node of interest and "unbundling"