New graph internal representation + visualization

[leclairm]

In order to not duplicate code and have less error sources from config to visualization, the graph visualization part should be implemented from the unrolled config directly. Doing it from the WorkGraph would get us closer to the actual executed workflow but we'd loose the cycles information (so the grouping of tasks) and would have to select what has to be represented.

Potentially we can think if it would be feasible to link it to the WorkGraph object so that we also have info about the status of tasks and data. I'm still unsure about that as the graph might be very big and pre-visualization would typically happen over a shorter simulation time than the actual production one.

[leclairm]

@agoscinski @GeigerJ2 @DropD I need your opinion on this before I start. I'm available for a call anytime today if needed.

After some thoughts, I have a new proposition which would go further than plugging in a visualization graph but rather refactor the intermediate representation (IR) part.

• We keep the parsing as is with config objects and pydantic, this is very useful.
• From the config objects, we generate spec objects, like TaksSpec, DataSpec, CycleSpec, TaskCycleSpec and DataCycleSpec. This is essentially renaming of the current objects.
• What is currently the unrolling mechanism generates the IR graph using the concretize() method from TaskSpec, DataSpec and CycleSpec taking the necessary TaskCycleSpec and DataCycleSpec as arguments and returning objects from the new Task, Data and Cycle classes. These would almost be the same as current unrolled objects but integrated in the graph structure (so core.Workflow would contain this graph as an attribute).
• From that graph we do the visualization
• From that graph we generated the AiiDA WorkGraph object

[agoscinski]

I think that should be fine, and if we want to create a visualization from the IR, it makes sense to resolve the dependencies in the IR rather than workgraph or visualization level individually. The only thought I had is that at the workgraph level we need to know if a dependency is created through input/output relationship or because it was specified by the depends/wait_on field in the yaml. So this need to be captured by this IR, even by keeping this in the tasks for give an edge a property.

[leclairm]

Yes sure. The task-task edge type must be captured. So I start working on it then. Hopefully I have stuff to show on Thursday ;-)

Thanks for the feedback