Advice: Implementing a "Vega Dataflow viewer" like visualisation using Deneb

[andrew-danieli-fj]

I'm looking for some advice on how best to implement a mind-map style visualisation similar to the "dataflow viewer" built into the vega/vega-lite online editor (e.g. rectangular nodes with text and directed arrows between them).

1. Would using the Directed Graph template work as the best starting point? I haven't been able to find any more appropriate examples than this and even it seems pretty complicated.
2. How would you switch the nodes from simple circle to rectangles with text content, and the links to arrows?
3. Given that the dataset coming in from Power BI is tabular, would I need to transform it internally in Deneb to match the nodes/links type structure used in the template, or would that transformation need to happen externally in Power BI?

Any pointers or simple examples would be a great help.

Thanks

[dm-p]

Hi there,

You are indeed asking for complicated examples to reference from, as node and edge data is particularly involved (especially in Power BI). I don't have the bandwidth to assist with language specifics, as I focus on developing and maintaining the tooling, so some of the locations detailed on Deneb's support page for seeking help may provide you with more readily available assistance. Some of those folks do periodically check this repo so I'll provide some quick starters and leave the issue open in case they're able to assist further.

1. If this is the closest analogue to what you're trying to build, then yes.
   • Alternatively, some of the folks linked on Deneb's community resources page have some quite advanced examples for Vega.
   • I'd suggest that you have a look at some of Davide Bacci's examples to see if they're either a better fit or there are parts of others you can adapt - notably the Gantt Chart and the Organization Tree Chart.
2. I might start by swapping them to a rect mark with the appropriate encodings (height, width, x, y).
   • However I personally haven't tested how force direction works with rectangular shapes, as the simulations are often geared towards circles in the examples (as well as any real-world visuals I've worked on).
   • To make a link an arrow, I'd suggest adding a symbol mark at the same location as the end of the link.
   • Again, Davide's Gantt chart example will show you how he did this for his work.
3. To make this work in Power BI with a single query (as that's what we're limited to), then you have a couple of options:
   • Use the standalone (non-certified) version of the visual and load the secondary dataset (likely the link data) from a remote location. I think this has been the course that the examples I've seen online for Deneb have taken.
   • Creative modelling of the data and transformation within the Vega spec to split into nodes and links. I personally haven't tried this for Deneb, but the Charticulator documentation has a guide for modelling the data to provide a tabular dataset that might work (as both Deneb and Charticulator use similar data view mappings from Power BI). Note that this relies on many to many relationships and may require you to produce a distinct set of results for the nodes based on the unique ID you assign it.

[andrew-danieli-fj]

Hi,

Thanks very much for the detailed response, and for producing such a great visual. I did some further digging and found a previous Deneb GitHub issue which highlighted the "1 query/dataset" limitation for the link between Power BI and certified visuals which I didn't know about. I am, for now, limited to certified visuals so will persevere with trying to transform the data into something suitable for Deneb/Vega to handle. It would be nice if Deneb could take a Json object as an alternative input so that the transformation of the data could be handled by Power BI. Like you I'm more a developer than visualiser and this is my first foray into that world.

[dm-p]

We couldn't unfortunately standardize an additional data role for a separate input, as it still has to be mapped to every row in the query by Power BI and this would perhaps create more overhead for the end-user in these cases.

That being said, I have personally, done this using a measure in the existing dataset and transformed it into something else (just taking the first row's field value and then working with that). I ended up using CSV data as it was easier to parse and TOJSON didn't work well with more than 10 rows, performance-wise. You may get some mileage from this approach if you can afford the performance hit on the query that Power BI runs.

We'd probably also need to think about how to validate when the data view is processed (as Power BI doesn't have JSON data types and we'd have to parse a string value to JSON) and structure this so that it's clear for templates what structure would be a valid input for any other consumers. We could probably dynamically generate a schema for this on the fly and make this part of the template but would be a breaking change for 1.x's template format.

I feel that given the current limitations of Power BI visuals, the 'correct' way would be to allow a conditional formatting property that could accept such a measure as an input. That would be treated as a scalar value by the visual and may have much less overhead. However, we would still have to think about how this would work in templates if it were to become a supported feature. We would need to at least (a) find a way to flag that an external measure dependency is needed in the visual properties and (b) that it should be a string that parses to valid JSON and match a particular schema for the template to validate correctly for the creator's intended use.

Anything that is going to have an effect on the template structure is slated for v2.0 in terms of when I can do the work, as it needs re-designing and may be a breaking change. I want to factor in how to manage additional model-based dependencies like drilldown and field parameters as this will require similar changes to the template format. I can keep this issue open to consider how this may work when the time comes, but I don't anticipate getting to this until later this year; I have 1.6 - 1.9 to get done first as these will contain most of the current asks outside of templating and will hopefully allow this templating work to be done in as targeted a manner as possible.

[dm-p]

Closing due to lack of activity. If you need further language assistance, I recommend approaching one of the community support representatives via one of their preferred channels.