Define overall architectural approach for exchanges

[cliftonc]

We need to agree high level principles and approaches for how data is exchanged.

It needs to be: Reliable, Scaleable, Developer Friendly, Secure & Future Proof.

Initial list of exchange patterns:

Type	Description
Notify / Pull	Notification via web-hook, followed by pull of data from source.
Publish / Subscribe	Publication of feed of changes on source, destination pulls data as needed.	Alternate/additive model to Notify / pull via event feed.
Push	Direct call of web service on destination from source.
Pull	Direct call of web service on source from destination.
Browser	Exchange via URL data in browser

[cliftonc]

• Define the output of the data and exchange work

[wvholland]

Perhaps its obvious for all off you and already taken into account but ... did we also consider resilience as one of the constaints. The whole construction should not collapse if something is not working. We will design it in such a way that we know that at some times things will fail, and still the system keeps working ...

[cliftonc]

The following was notes from a conversation between @mcginkel @EvanderVeen and @cliftonc over the previous 2hrs.

Principles / Areas

• Timing - "real time" vs nightly / batch.
• Async vs Sync - mix of model, with focus on async notification to avoid nightly batch - synchronous as a fallback.
• UWLR - data sync in bulk; student added to group during the day - can't use LAs - this is an anti-pattern we need to avoid.
• If data doesn't exist that you need then give meaningful messages, or go and get the data you need
• Current standards assume certain data is always present, and exactly what you get - so not flexible.
• Assume minimum data - all other is optional? How do we deal with different consumers needing different data sets?
• Profiles related to Oauth flow - e.g. request data access that is approved - via scopes with recognisable names [talk in security session] - scopes should be connected to API design, e.g. separation of student / student address.
• Define needs / triggers for data processing agreements?
• Personas - e.g. LA for one subject / project vs larger platform
• Existing standards - lets be pragmatic and assess with each exchange

Patterns for integration

• Push changes
  • No polling - except as fall back
  • If you miss events you can come back and get them
  • Should the events contain data - this requires further discussion on pros/cons / are there rules?
• Get data when needed
  • How do we deal with batch / bulk data efficiently?
• Rate limiting / fair use
  • Service providers can define terms independently
  • How do we model relationships for more complex objects (groups <> students)

Approach to identifiers:

• Each providing system has a responsibility of providing a unique identifier for each data entity
• In some scenarios - e.g. learners - there may be additional 'external' identifiers such as the ECK ID
• Event feed should always reference that identifier in any communication

Proposal for Exchanges:

• Three parts:
  • Webhook subscription for 'live' events
    • Need to provide a way for subscriber to confirm message came from right source (e.g. https://developer.github.com/webhooks/securing/)
  • API to get past events (like an event stream - paginated)
    • Need to define the format and structure of the event stream
    • What is an event? Business or data event? - more likely to be data events 'user updated'.
  • API to get specific objects referenced in events
    • Rest as default
    • Graphql is useful here as it allows you to 'reach into' the tree of relationships

Examples from other places to explore

• Stripe https://stripe.com/docs/api/events
• Github https://developer.github.com/v3/activity/events/
• Twilio https://www.twilio.com/docs/usage/monitor-events
• https://www.asyncapi.com/ may be interesting to document along with https://www.openapis.org/

[EdwinIddinkGroup]

Security and data transmission must meet the standards as laid down in Edu-K / Edustandaard. This includes the use of / taking into account:

1. • The Edukoppeling transaction standard is part of the reference architecture for education (ROSA). Within which the use of REST and SOAP is guaranteed.
2. • Information Security and Privacy ROSA Certification Scheme.

[EdwinIddinkGroup]

Patterns for integration: Push changes

• No polling - except as fall back. Get data when needed If I place these two good starting points within the starting point: you only get the data that you need at that moment based on goal binding. (data minimization) What is wrong with real-time retrieval? Please note that the "license" and therefore the information about the "link" is currently being retrieved in real time (currently within VO 3.5 million).

Okay I admit the information about a license changed in a school year but 3x (not yet used, in use and expired / canceled / blocked) and an access link will not change very often either. But in an N on N relationship to external systems, I wonder if the push isn't getting more complicated.

If I now look at the current development around data minimization, in which the surname may only be shown in an application if this is also necessary in relation to the position, for example: entering numbers / absence: two students with the same first name, in the same group only for those two students the last name is shown, then I do not see push in relation to a notification message works. By the way, as an additional requirement for this type of data the information is not aloud be saved (only available during the session and then in memory (?))).

[cliftonc]

Thanks @EdwinIddinkGroup - can you link to the appropriate versions of these so we can all review and understand if / how we would like to incorporate them?

We discussed existing standards, and agreed to look at them related to each exchange and see if they met our principles and were suitable to take forward, so it would be helpful if you can collect and link the ones you feel are relevant.

You are (purposely I assume?) using some strong language - must - can you also share details on why that is? Is there a legal requirement or other binding agreement we need to take into account that the group isn't aware of?

[cliftonc]

Re. The comment on push Vs pull.

There are three elements to the design, a real time web hook, an API that can be polled for events, and the API for a specific data item.

A subscriber may ignore the first two and just call the API when needed (this is your example), but choosing to do this may have impacts on performance or reliability, and be at risk of being rate limited by the provider if this is not expected behaviour.

If we want it to be reliable, you would use all three components.

SOAP isn't designing for the future, we agreed to let it enjoy its retirement, along with files over SFTP.

Re. The comment on data:

We all agree whole heartedly to data minimisation and it will be a fundamental part of the architecture, but perhaps a key element is putting it in the control of the school / customer. For example, using oauth scopes we can ensure that a subscriber can only receive basic data about users (minimal to operate the service), but not address details if they aren't sending physical books.

This is a dramatic improvement over UWLR where a lot of data is pushed to everyone.

What does concern me is making arbitrary decisions like 'product A can never get that data because committee B decided that it wasn't needed for it to function'.

If data is required for a feature that a school is paying for, and the school approves that data exchange (along with any necessary processing agreements), there should be no issues with this. All members in the ecosystem are bound by the same legal requirements related to data protection and privacy, we shouldn't create additional layers over the top.

If it isn't done this way we will stifle innovation in the sector, and for no actual benefit to the school or individual that I can see.

Does this need a specific issue and sub-group to discuss and clarify? Perhaps you can lead it and kick it off with a review of current agreements that will be binding for us all?