search

cms-gem-daq-project / ctp7_modules

0 stars 13 forks source link

[RFC] Handling large data transfers in the RPC framework #168

Closed jsturdy closed 4 years ago

jsturdy commented 4 years ago

Brief summary of issue

As per discussion on #167

I agree that the message size should not matter to the user and should be, if required, fragmented in the lower layers.

However, this is not how the rpcsvc framework/service is designed; the messages must be kept relatively small, under 8 MiB. You could probably work around it in the templated RPC framework; it would probably be easier and faster to re-write the rpcsvc daemon. Also, the RPC calls are not designed as a streaming service, so that the whole message must be buffered at some point... but the memory is limited in the CTP7.

_Originally posted by @lpetre-ulb in https://github.com/_render_node/MDE3OlB1bGxSZXF1ZXN0UmV2aWV3MzQ4NTY3Njc4/pull_request_reviews/more_threads_

Types of issue

Context (for feature requests)

Passing large chunks of data between the back-end and the PC is a possible use case that we need to have a solution for (now, and potentially more importantly for an upgraded back-end). @lmoureaux, if we sent a full AMCs worth of configuration data, how much data are we currently talking?

lmoureaux commented 4 years ago

Is this really a high priority item? It doesn't seem critical for the next MWGR.

Neither the rpcsvc service nor the template RPC frameworks were designed for large data transfers. They require the whole dataset to be available before sending it.

An option to implement this feature would be to open a socket, allowing the two ends of a call to communicate asynchronously while the call is running. This could be achieved by constructing a special data type whose serialization and deserialization hooks would handle the socket. It would be passed as function arguments. It would not be trivial to implement and use.

The size of the configuration is driven by VFAT data. Each channel is 2 bytes, plus 18x4 bytes for control registers. Therefore an AMC configuration is O(100kB).

jsturdy commented 4 years ago

OK, good, if the configuration data is not even close to the rpcsvc limit, then I agree, this is not a high priority item

lpetre-ulb commented 4 years ago

Moved to the new GitLab repository: cmgemos#87.