Alignment of I/O interfaces and data types for data processing workflow

[s-minoo]

Pitch

Current data processing components have different configurations for the same I/O type (websockets, http, etc..). Furthermore, the representation of the data types is also different across the different processing components. A generic I/O interface and data types, independent from the different languages, would enable a strong foundation to a more modular data processing workflow architecture without reliance on existing tech stack on data processing.

Desired solution

• A set of ontologies to describe the I/O interfaces and data types
  • Could be an extension of RML's Logical Source/Target
• A set of I/O interfaces configured using the previously defined ontologies
  • An FFI would unify the interface for all different languages (Rust seems to be a candidate)
• A generic set of interfaces covering all possible data types with ease of extension
  • A wrapper which encode the messages to strings/bytes as desired by the consuming/generating component

Acceptance criteria

• A library to interpret the defined set of ontologies and create the I/O interfaces which are exposed to the data processing application developers
• A simple Java library which could be reused when building data processing application with different frameworks.
  • For example, an application in Flink or Spark could use the library's interfaces to build their own I/O connectors.
• A working data processing workflow application with two different processing frameworks communicating with each other using the implemented Java library

[bjdmeest]

Rephrasing this a bit, to make the link more explicit, and removing the data type parts because that could be a next step

Pitch

Streaming (RDF) data can be accessed through various different interfaces (websockets, http, etc..), whilst current data processing frameworks (e.g. Apache Flink, NiFi, Spark, ...) have different configurations for the same interface. With a generic I/O interface that can be applied to different frameworks, we provide an abstraction layer that makes it easier to onboard existing data onto Solid pods. Furthermore, this would enable a strong foundation to a more modular data processing workflow architecture without reliance on existing tech stack on data processing.

Desired solution

• A model and interface to describe the I/O interfaces and data types
  • Could be an extension of RML's Logical Source/Target
• A set of implementations following this interface to showcase its generic applicability
  • A Foreign Function Interface would unify the interface for all different languages (Rust seems to be a candidate)

Acceptance criteria

• An interface specification that can solve multipe streaming modes (e.g., pull vs push, unicast vs broadcast, etc.)
• Two test cases that use the same easy interface to extract RDF data from a streaming data interface, each using a different framework under the hood (e.g. getting RDF data from Flink, and from Spark)
• A working data processing workflow application with two different processing frameworks communicating with each other using a single interface design

[RubenVerborgh]

This is a good technical challenge!

In the context of this repository, we try to be pin things to a specific use case, such that there can be a specific demonstrator. (See for instance #1.)

Are there use cases available that would be a good match, and can we apply them?

[pheyvaer]

@bjdmeest @s-minoo What does "FFI" stand for?

[bjdmeest]

@bjdmeest @s-minoo What does "FFI" stand for?

https://en.wikipedia.org/wiki/Foreign_function_interface , I updated the description

[pheyvaer]

@s-minoo Did you have the chance to look into making the necessary changes?

[s-minoo]

This one has evolved into #47 where there's also a need of I/O alignment.

51 will contain the ontologies to align the I/O interfaces.

[pheyvaer]

Does that mean that this one is no longer needed?

[s-minoo]

I think it's better to close this one. Since #51 will eventually be the end result of this challenge.