ENH: callback to send data across the wire

[CJ-Wright]

It would be nice to have the capacity to send data and metadata across the wire live.

Expected Behavior

1. Point callback to server somewhere and a file folder on server (for tokenized data)
2. Subscribe to RE
3. Run experiment
4. As experiment is running metadata is being sent to server.
5. When experiment stops (either on stop doc, or new start if halted?) send tokenized data (in its original form) over to server with its tokens so that they can be linked on remote server.

Current Behavior

What happens (live) at the NSLS-II stays at the NSLS-II.

Possible Solution

I presume that the metadata/json looking things get sent over via a RESTful API. I'm less certain about the whole token/large data on the wire aspect.

[prjemian]

We're looking at making a callback to emit the event documents through 0mq. Something like that what you have in mind?

[prjemian]

re-phrasing: What happens at the beamline could be shared with HPC.

[CJ-Wright]

Isn't there already a ZMQ driven callback?

[prjemian]

Can't keep up, it seems

[danielballan]

What happens at the beamline could be shared with HPC.

Agreed!

@prjemian There is this, which we took for a spin during my visit to APS. It serializes documents into a custom (dead simple) serialization format and broadcasts them to N subscribers. It's a working proof of concept; a long term solution might use an existing serialization format and might use something heavier than 0MQ.

@CJ-Wright Could what you propose be achieved by a script that uses our Publisher? I'm not sure what to make of "It would be nice to have the capacity to send data and metadata across the wire live," because the 0MQ proof-of-concept -- which I know you have built on for XPD -- does that.

[CJ-Wright]

At XPD we de-tokenize on the client side. I guess we could de-tokenize on the server side? We may not have assetstore on the client side so the tokens don't mean much.