Rich-endpoint V1

[mfahdaz]

Aspect	Detail
Summary	A unified interface to transport the diverse nature of data.
Task Area	Design
Assignee	Wouter Klijn
Information
Prerequisites
Dependencies	Almost all other task

Summary

The rich-endpoint as a concept sees functionality to be integrated into simulation applications running on the companion app. This assures that, if this companion app is tied to core zero, the processing on the main theads are not slowed down.

The functionality will be exposed to the outside world via services. That will transport the C&C command via a unified zeromq transport layer. Rich endpoints is a concept with out a specific direct class of object to point to. It is implemented in: companion app communication protocol (secure communication e.g. using PKI) orchestrator services exposing the functionality

Tasks

• [x] Find correct location to document the requirements as detailed in the copy past of Fahad initial information in this issue
• [x] Create a diagram with the Rich endpoint in the center with all the different components which relate to it
• [x] Define the requirements: roles / functions
• [x] Rough class diagram and functions (arguments)

Clearly wouter needs to work here

Requirements

Acceptance criteria

[w-klijn]

Copy past of the data collected by Fahad

• Plugin
• Services
• API

A unified interface to transport the diverse nature of data.

Communication types:

• Command & Control
• HPC communication

Known variations of data to be transported:

• Simulation data
• Visualization
• Steering
• Monitoring information
• Provenance data

Requirements

• The system must be able to receive spikes from a simulator running on multiple nodes.
• The system must be able to sort the incoming spikes into spikes per neuron, transpose the data.
• The system must be able to format the outgoing spikes into data objects that can be injected directly into the translation software.
• The system must be able to collect the output of the translation software and forward this to a second simulator.
• The components in the system must be able to run concurrently.
• The data from simulation should be transferred as fast as possible
• The amount of data sent per time step to analysis could be flexible
• The user could be able to request data flexible, at different rates and times
• The system must be able to exchange of parameters and settings before the simulation, and data transfer starts
• Simulation Information about the start and the end of the simulation must be exchanged between simulation and analysis.
• The received data should be stored in a buffer. This is necessary to allow intermediate processing and transfer in chunks of data rather than a 1:1 mapping from incoming to outgoing package sizes.
• The framework should enable data access at arbitrary rates and asynchronous.

[w-klijn]

The co-simulation framework has a large number of functions to prevent repeated work and complicated integration. The Fat-end points will be the single API / plugin that combines the different needs. Between the shared functionality and the different application a thin shim will be needed to enable integration and to allow separation of concerns.

Expected functionality per Fat-end point: Communication:

spike/rate data exchange
Simulator state needed for network generation
Steering commands
MPI configuration
Basic monitoring information

Tasks

[mfahdaz]

estimated TBD by 11.09.2021

[mfahdaz]

Rough design diagram can be found here. Later, it will be moved to design document located at HBP Wiki drive.