Global time-centric data inspector

jonnor commented 10 years ago

Right now it is very hard to determine the order of events flowing through a graph, and to see the casual relationship between events. It would be very useful to see the data ordered by time, and then by edge. One view could be a global edge packet log, similarly to current edge inspectors. Another view could be a timeline, with multiple tracks - one track for each edge viewed. This should have visualizations for common datatypes: scalars, booleans, colors, strings etc. For both views it should be possible to press the data points to see it in more detail. It should also be possible to register custom visualization plugins.

This would require having timestamps, as well as monotonic sequence numbers on edge data provided by the runtimes.

automata commented 10 years ago

Sounds related with #151

bergie commented 10 years ago

Related: #245 and #243

jonnor commented 7 years ago

Notes from brainstorming with @bergie about this topic today

Aside: Input packets creators

Creating/sending input packets is a 'dual'/'inverse' of packet inspection (receiving/viewing). Right now lives in the "node inspector". It should be possible to implement packet inspection improvements without changing the existing packet creators.

Improvements for input packets thus considered a separate, independent project(s).

Some of the ideas may however have strong similarities:

Functionality provided in a single panel, accessible regardless of graph/navigation level
Panel has a set of controls (instead of one panel per input port)
Abitity to select/expose a 'control' for a given input port. Persistent workspace state
Each control can have an extended/detailed widget (maybe under expander?)
Possibility of custom/plugin control widgets

Milestone 1. Persistent edge subscriptions

Stored in memory as state

Managing subscriptions

A way to see list of current subscriptions,
add an edge to subcription,
remove an edge from subscriptions
List goes into timeline area

BONUS: highlight mapping between subscription listing entries and corresponding edges in graph

Edge inspectors

Move down to timeline area.
Still shows all data for an edge, scrollable

Desired: Persistent selection of edge subscription decides inspectors to show Minimum: 1-1 mapping between edge subscription and edge inspector

Milestone 2. Timeline

"Edge inspector" becomes "Payload inspector"

Shows only a single packet, not whole history

Timeline widget

Shows box where packets occurred, for each subscribed edge
Can zoom in-out to show different time ranges
Can scroll back-forth to shift viewed time range
Can select packets to view in payload inspectors

Automatic scrolling/selection

Default: Follows as data comes in Timeline scrolls to the side to show it, and payload inspector moves to the latest packet
If manual selected packet and/or timeline is viewing the past, do not automatically scroll

Milestone 3. Searchable/filterable

Payload inspector

Can filter/select data. Query is persisted. For ex using a JSONPath

Timeline

Can select/highlight packets on matching a (JS) predicate
Maybe Regexp and JSONPath

?? Predicate only per-packet. Or seaching in matching

Later

Packets persistance

Saved Locally in browser. Saved in-cloud. Syncronization between them

Payload visualization plugins

Allows to extend Flowhub with packet format specific visualizations. Our own 'payload inspectors' should use this mechanism

User-providable plugins? Third-party-providable plugins?

Payload diff visualisers

Instead of visualizing one packet, takes two packets and shows the difference between them. Especially useful for comparisons. Either on two different edges (ex: before/after component)

Also very relevant for tests (expectation versus actual).

Timeline visualizations plugins

Can make plugins for rendering data over (matched), sets of packets in the timeline. For instance to render a graph of number varying across packets, or color changing

May need to be able to specify a transformation (JSONPath etc) to map/extract raw edge/payload data to the value to visualize.

Flowtrace integrated

The data (and system/graphs) to inspect can come from a loaded Flowtrace blob Possibly from far in the past Can possibly also be streamed in. Maybe done via a pseudo-runtime adapter??

Persisted workspace state

Can save all the state: which edges are, timeline selection, payload visualizers/queries. For instance to a .json file that can be stored in Github as part of project, Flowhub can open this too, allowing going back to commonly used view of a system.

Can send a URL to a collaborator for a project (or runtime/flowtrace) including (reference to) the workspace configuration. Upon opening, the person gets exactly the same UI view as you.

bergie commented 6 years ago

Marble Diagrams might be an interesting way to show this.

Some examples:

http://rxmarbles.com/ https://rxviz.com/

bergie commented 3 years ago

Basics are now done via #1039. We should make the further enhancements tickets of their own

noflo / noflo-ui