ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

Modelling of Bifrost graphs with Dataflow Interchange Format #26

Closed MilesCranmer closed 7 years ago

MilesCranmer commented 8 years ago

See this paper for a description of the format. It was created in order to solve the "critical problem [which] arrises in transferring a DSP design across dataflow-based tools due to the lack of a vendor independent language for specifying DSP-oriented dataflow graphs." (An example is shown below).

Another paper describes compiling these graphs into C, using different schedulers and memory management tools to help optimize the software. In section 4.2, they describe their use of ring buffers: @benbarsdell - I am curious to hear how much the Bifrost implementation differs from their's, because they provide a good explanation which may be useful in creating documentation for Bifrost.

[keyword] graph graphID [basedon graphID] {
    params {
        param prm1, prm2, ...;
        domain (prm1, {1, 2, ...});
        domain (prm2, [1, 5]);
        ...
    }
    interface {
        input portID portID ...;
        output portID portID ...;
    }
    topology {
        nodes { nodeID [:portID ] nodeID [:portID ] ...}
        edges {
            edgeID sourceINodeID sinkNodeID;
            edgeID sourceINodeID sinkNodeID;
        ...
        }
    }
    refinement {
        subgraphID nodeID
        subPortID:edgeID subPortID:PortID ...;
        subgraphID nodeID
        subPortID:portID subPortID:edgeID ...;
        ...
    }
    attribute attributeName {
        edgeID value ;
        nodeID value ;
    ...
    }
    ...
    [built-in attribute] {...}
    ...
}

In addition, here is an interesting chart which shows how DIF is incorporated into programming models:

DIF

benbarsdell commented 8 years ago

Thanks Miles. I had a glance at this last night too.

I have to admit that my first impression was not altogether positive:

The language used in Section 4.2 is a bit foreign but I think they're describing the same thing as what I called the 'ghost region' that's used to emulate wrap-around access.

I don't want to dismiss this, as it may well prove useful, but I would probably treat it similarly to GRC, i.e., a pipeline/graph serialization format that we should consider supporting.

On 8 July 2016 at 14:58, Miles Cranmer notifications@github.com wrote:

See this paper http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.76.6859&rep=rep1&type=pdf for a description of the format. It was created in order to solve the "critical problem [which] arrises in transferring a DSP design across dataflow-based tools due to the lack of a vendor independent language for specifying DSP-oriented dataflow graphs." (An example is shown below).

Another paper http://www.ece.umd.edu/DSPCAD/papers/hsu2005x2.pdf describes compiling these graphs into C, using different schedulers and memory management tools to help optimize the software. In section 4.2, they describe their use of ring buffers: @benbarsdell https://github.com/benbarsdell - I am curious to hear how much the Bifrost implementation differs from their's, because they provide a good explanation which may be useful in creating documentation for Bifrost.

[keyword] graph graphID [basedon graphID] { params { param prm1, prm2, ...; domain (prm1, {1, 2, ...}); domain (prm2, [1, 5]); ... } interface { input portID portID ...; output portID portID ...; } topology { nodes { nodeID [:portID ] nodeID [:portID ] ...} edges { edgeID sourceINodeID sinkNodeID; edgeID sourceINodeID sinkNodeID; ... } } refinement { subgraphID nodeID subPortID:edgeID subPortID:PortID ...; subgraphID nodeID subPortID:portID subPortID:edgeID ...; ... } attribute attributeName { edgeID value ; nodeID value ; ... } ... [built-in attribute] {...} ... }

In addition, here is an interesting chart which shows how DIF is incorporated into programming models:

[image: DIF] https://camo.githubusercontent.com/5134f94a3624940988b9e1624df436d0dab301b4/687474703a2f2f692e696d6775722e636f6d2f717a4f4366776c2e706e67

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ledatelescope/bifrost/issues/26, or mute the thread https://github.com/notifications/unsubscribe/ADy3WDVdjAr8akBjUk0deuku2OTHUJm9ks5qTsfxgaJpZM4JIbk3 .

MilesCranmer commented 8 years ago

Sorry for Figure 1 - it does indeed look a bit scary now that it's enlarged on my screen. Also, I don't think I was completely clear about what "Modelling of Bifrost graphs with Dataflow Interchange Format" meant - I mean what you say - just making a converter between DIF and Bifrost pipelines (DIF as the serialization format). I prefer DIF over GRC's format because DIF has converters between many other formats, so we might as well just support DIF from the start. There is also a paper which discusses integrating GNU Radio with DIF, by the creator of GNU Radio.

The second paper is a bit less abstract about what everything means - they give a few examples with reference flow graphs (e.g., Figure 4). The language reference manual also has full explanations for all the terms used.

Does the 8 year number refer to the software that they have for DIF on their website? The Maryland group is still publishing research related to DIF, and there are many different programs out there that use it. The format is definitely still in use and I think it's the most broadly used of its kind.

ledatelescope commented 8 years ago

I would like us to look widely (proportional to the time available) but be conservative in what we adopt. Keep it simple and ensure connection to an extant community. It would be interesting to learn what Jack Hickish or Dave MacMahon may have come across. Danny?

Lincoln

On 7/9/16 06:20, Miles Cranmer wrote:

Sorry for Figure 1 - it does indeed look a bit scary now that it's enlarged on my screen. Also, I don't think I was completely clear about what "Modelling of Bifrost graphs with Dataflow Interchange Format" meant - I mean what you say - just making a converter between DIF and Bifrost pipelines (DIF as the serialization format). I prefer DIF over GRC's format because DIF has converters between many other formats, so we might as well just support DIF from the start. There is also a paper which discusses integrating GNU Radio with DIF http://www.ece.umd.edu/DSPCAD/papers/zaki2009x1.pdf, by the creator of GNU Radio.

The second paper is a bit less abstract about what everything means - they give a few examples with reference flow graphs (e.g., Figure 4). The language reference manual http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.81.1863&rep=rep1&type=pdf also has full explanations for all the terms used.

Does the 8 year number refer to the software that they have for DIF on their website? The Maryland group is still publishing research related to DIF, and there are many different programs out there that use it. The format is definitely still in use and I think it's the most broadly used of its kind.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ledatelescope/bifrost/issues/26#issuecomment-231504340, or mute the thread https://github.com/notifications/unsubscribe/AJfeMoBMJKHE7jiUJO65m9iYiNd5kQ8xks5qTvBpgaJpZM4JIbk3.

Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX: 1 617-495-7345 greenhill@cfa.harvard.edu Skype: ljgreenhill www.cfa.harvard.edu/~lincoln

MilesCranmer commented 8 years ago

Here are some other options:

Something that is also important to consider: how straightforward it is to map between main Bifrost concepts (#23) and the graph description. For example, given the current API usage, I think it might be easier to translate the simpler DOT format into python code.

Lincoln brought up the good point that it's also important to consider what community we want to target for using Bifrost. If we support DIF, we might attract that research community, and if we support GRC's format, it might be easier to draw in the software radio community. Would there be an advantage to supporting MDL (i.e., Simulink) format?

ledatelescope commented 8 years ago

Why not go with NoFlo? Pros/cons are unclear to me.

On July 9, 2016 10:42:36 PM GMT+05:30, Miles Cranmer notifications@github.com wrote:

Here are some other options:

  • (Wiki list) General graph representations
  • The DOT language I know is very popular, and is used by Doxygen to generate inheritance graphs, among many other visualization tools. This actually makes me wonder if there is already tools to save a NoFlo design in DOT. The limitation is that location of each node on a visual interface can't be specified easily (?), so we might have to rely on NoFlo's autolayout tool if you are loading in a graph that you programmed with another interface.
  • Other XML-based and text-based languages are given that I haven't read into too much yet.
  • (Wiki list) Dataflow programming languages
    • Many of these languages have their own specific formats.
  • It is interesting to note that @hughbg is our own resident expert on SISAL, if anybody comes across it. SISAL uses the "IF1" for graph descriptions, which is a language for programming dataflow hardware

Something that is also important to consider: how straightforward it is to map between main Bifrost concepts (#23) and the graph description. For example, given the current API usage, I think it might be easier to translate the simpler DOT format into python code.

Lincoln brought up the good point that it's also important to consider what community we want to target for using Bifrost. If we support DIF, we might attract that research community, and if we support GRC's format, it might be easier to draw in the software radio community. Would there be an advantage to supporting MDL (i.e., Simulink) format?


You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/ledatelescope/bifrost/issues/26#issuecomment-231544846

Sent from Droid Turbo. Please excuse use of typos and abbreviations.

ledatelescope commented 8 years ago

BTW, this research is really great. Good work!

On July 9, 2016 10:42:36 PM GMT+05:30, Miles Cranmer notifications@github.com wrote:

Here are some other options:

  • (Wiki list) General graph representations
  • The DOT language I know is very popular, and is used by Doxygen to generate inheritance graphs, among many other visualization tools. This actually makes me wonder if there is already tools to save a NoFlo design in DOT. The limitation is that location of each node on a visual interface can't be specified easily (?), so we might have to rely on NoFlo's autolayout tool if you are loading in a graph that you programmed with another interface.
  • Other XML-based and text-based languages are given that I haven't read into too much yet.
  • (Wiki list) Dataflow programming languages
    • Many of these languages have their own specific formats.
  • It is interesting to note that @hughbg is our own resident expert on SISAL, if anybody comes across it. SISAL uses the "IF1" for graph descriptions, which is a language for programming dataflow hardware

Something that is also important to consider: how straightforward it is to map between main Bifrost concepts (#23) and the graph description. For example, given the current API usage, I think it might be easier to translate the simpler DOT format into python code.

Lincoln brought up the good point that it's also important to consider what community we want to target for using Bifrost. If we support DIF, we might attract that research community, and if we support GRC's format, it might be easier to draw in the software radio community. Would there be an advantage to supporting MDL (i.e., Simulink) format?


You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/ledatelescope/bifrost/issues/26#issuecomment-231544846

Sent from Droid Turbo. Please excuse use of typos and abbreviations.

telegraphic commented 8 years ago

This is really exciting stuff, but also looks like a deep, dark rabbit hole that leads to a looking glass that leads to a stargate that leads to a tear in spacetime in which Matthew Mconnaughey is stuck in an open singularity tesseract thing sending signals to his child through gravity waves who will end up being older than him thanks to relativitstic time dilation. That is: I think it's a time sink.

As much as I enjoyed interstellar, I advocate focusing on core functionality, documentation, and a really nice python API before heading too deep. But certainly we should ensure the APIs are compatible with dataflow descriptor languages.

ledatelescope commented 8 years ago

Developing the same weirdly similar concern myself.

On 7/11/16 12:04, Danny Price wrote:

This is really exciting stuff, but also looks like a deep, dark rabbit hole that leads to a looking glass that leads to a stargate that leads to a tear in spacetime in which Matthew Mconnaughey is stuck in an open singularity tesseract thing sending signals to his child through gravity waves who will end up being older than him thanks to relativitstic time dilation. That is: I think it's a time sink.

As much as I enjoyed interstellar, I advocate focusing on core functionality, documentation, and a really nice python API before heading too deep. But certainly we should ensure the APIs are compatible with dataflow descriptor languages.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ledatelescope/bifrost/issues/26#issuecomment-231652706, or mute the thread https://github.com/notifications/unsubscribe/AJfeMrwE3uMTCwX2pyHJdTsNg3i0ahPZks5qUePigaJpZM4JIbk3.

Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX: 1 617-495-7345 greenhill@cfa.harvard.edu Skype: ljgreenhill www.cfa.harvard.edu/~lincoln

MilesCranmer commented 8 years ago

Good point. I will leave this until we have some more blocks set up and optimized.