search

microsoft / Data-Appliance-GX

Sovereign Data Sharing Appliance/Connector for enterprise scenarios
MIT License
8 stars 7 forks source link

Demo protocol - continuing work #85

Closed jimmarino closed 3 years ago

jimmarino commented 3 years ago

Getting started

This PR is for discussion and not intended for merging yet.

The purpose of the demo protocol is to provide a simple data transfer extension that can be used to:

  1. Try out and learn about the internals of the runtime without having to perform a complex setup
  2. Serve as a testing ground for different data flows and use cases, for example, streaming data scenarios
  3. Serve as a data transfer extension that can be used in automated testing

To get started, run DemoPushStreamTransferTest. Only the web socket push streaming scenario is currently implemented (see below).

Longer-term idea

I think there may be value in abstracting some of the capabilities into this extension into more general, production-grade extensions:

  1. We may want to add the ability to publish web socket endpoints in the web extension similar to the way JAX-RS works

  2. We may want to have an extension (or extensions) that supports web socket and webhook-based messaging

Regarding point 2, a websocket/webhook data transfer extension could support a use case where "event data" is transferred via a dataspace. In this case, the provider connector transfer extension could receive a request for an event stream and forward it (with access credentials and an endpoint URL) to another system, which would begin publishing a stream. On the consumer connector side, the transfer extension would have setup a webhook or websocket endpoint. This endpoint would be outside the controller process and could be a lambda or some other infrastructure that receives events. If the events are JSON, they can be dispatched by their polymorphic type to a handling system.

What's implemented and what could be added

I think there are a number of use cases the demo extension can implement:

  1. Push streaming over web sockets. The consuming connector sets up a destination that the providing connector publishes to over a web socket. This is implemented in DemoPushStreamTransferTest.

  2. Push streaming over HTTP. This is not yet implemented but could be done by creating an HTTP StreamSession implementation and JAX-RS endpoint to receive messages.

  3. Object storage over HTTP. The consuming connector provisions a storage container and sends a request to the providing connector which sends the data over HTTP. This has not been implemented and could be a good thing for someone to pick up.

  4. Pull streaming over web sockets. The consuming connector sends a request to the provider, which provisions an endpoint destination the consuming connector can subscribe to and receive data.

  5. Pull streaming over HTTP. I'm not sure we want to do this, but for completeness, we can list it. Basically, scenario 4 but over some form of HTTP long-polling.

ghost commented 3 years ago

CLA assistant check
All CLA requirements met.

github-actions[bot] commented 3 years ago

Unit Test Results

  43 files  ±0    43 suites  ±0   5s :stopwatch: ±0s 104 tests ±0  100 :heavy_check_mark: ±0  4 :zzz: ±0  0 :x: ±0 

Results for commit b6ce3adf. ± Comparison against base commit b6ce3adf.

:recycle: This comment has been updated with latest results.