Flowtrace format

[bergie]

Just like traditional programming environments have stack traces and core dumps, it would be useful to come up with a file format to store the state of a FBP network.

The relevant information in the file would be:

• Metadata dictionary about the environment (runtime type, versions, possible env vars, etc)
• The current graph(s) being run in the network (in the standard JSON format)
• The packets that have recently been transmitted (in the packet payload format of FBP protocol)

These files could be generated on demand from tools like Flowhub, or automatically by FBP runtimes themselves whenever an error occurs.

Various Flow-Based Programming tools could be made capable of opening these files to show the graph and the packets.

[jonnor]

Packets payload format should be extended to include timestamp and sequence number

[jonnor]

Would we want to allow changes to the graph over time of one trace? My hunch is to say no, and require that to become a new trace. It might however be useful to be able to have multiple traces in one file/stream.

[jonnor]

I see the environment and the graphs info as being part of the header of the file, and then the contents of packets is the body. The header would not change after tracing started, and we could then stream/append for the rest of the file. If the format requires some sort of tail (like JSON would), it would be good to have a fixing tool which can add this tail in case of stream being uncleanly terminated.

[jonnor]

Given that there will be a lot of data in the packets, we could probably standardize on some sort of compression for these files. For instance gzip.

[jonnor]

Work is ongoing in: https://github.com/flowbased/flowtrace