Adapting data_feed server to handle all OCS feeds

BrianJKoopman commented 5 years ago

As mentioned in #38, there's been some discussion about enabling the data_feed server to handle all OCS feeds. There are a couple of points to bring up for discussion.

Getting a list of all available feeds needs to happen automatically. I believe this list is available via an RPC to an OCS function. OCS's clients are also able to get a list of the feeds. It was suggested we use this functionality provided by an OCS client, however this would require making an OCS client within the running sisock server, which adds OCS as a dependency. I think it would be better to figure out the address for the RPC and make it ourselves, much like we are doing with the pub/sub subscription address.
Moving all data feeds to a single server simplifies the configuration for live monitor, only one container is needed for live monitoring all OCS Agents which publish data. This does sound preferable, however, if this data server needs to be restarted for any reason, the hour cache is lost for all data. Separate servers avoids this in the obvious way. However it makes me wonder if we can improve the way we are storing the cache instead, so that a data_feed server could be restarted without losing the cache.

jlashner commented 5 years ago

Sorry I may have not been clear on slack, but the RPC I was talking about was to get all feeds for a particular agent. You can do this by calling observatory.instance-id and passing it the string get_feeds as an argument.

If you want a single container for all agents, I'd recommend using the registry to get feed info for all registered agents. This is what the aggregator does to figure out which feeds to subscribe to.

BrianJKoopman commented 5 years ago

Ah, I see, thanks for the clarification Jack. Are there agents that publish data to multiple feeds yet?

To get the feed info for all registered agents we'd need to subscribe to the "agent_activity" feed and have a client call "dump_agent_info"?

jlashner commented 5 years ago

I don't think there are currently any agents that have more than one feed, but maybe the Lakeshore372 agent might want a separate feed for temperatures and heater outputs... For the load curves I just put them in the same feed.

Yeah, that'll do it.

BrianJKoopman commented 5 years ago

Pattern based subscriptions is likely how this should be implemented. The structure of the cache might have to be adjusted.

simonsobs / sisock

Adapting data_feed server to handle all OCS feeds #39