Add support for distributed Cameras

[AnthonyHorton]

Experience with the Huntsman Telescope has shown that running arrays of USB cameras from a single computer is unreliable. Consequently the Huntsman team have decided to follow the example of the Dragonfly Telescope and change to a distributed control system, with each camera being controlled by its own computer and a central control computer performing the observatory level tasks.

This requires additional features for POCS to support distributed operation. Fortunately the modular, object orientated structure of POCS provides an natural interface for the split, that between the Observatory object and the Camera object. What is required is a network layer between these objects.

[jamessynge]

I agree with the idea of moving to a more distributed setup; it is compatible with indilib and indigo, as well as my desire to have each of the Canon cameras controlled by its own Raspberry Pi Zero W (for sites that allow WiFi). However, this particular approach (Pyro) being taken is an alternative to indilib and indigo, rather than complementary.

I'm also interested in making POCS more robust, and am not sure how this effort relates. For example, the scheduler regularly gets confused near the end of the night, which locks up POCS; and I've seen the iOptron driver get confused, and thus it thought a single slew was occurring for a day and a half. While we can fix these individual bugs, splitting the system up so that there are actor processes and supervisory processes would help us deal with the inevitable future bugs. For example, the supervisor could have a rule that says that any one action (slew to coords or start tracking) taken by the iOptron mount should take no more than 1 minute.

Before proceeding with addressing this issue, can we make sure that we're going in a direction that works for PANOPTES?

[AnthonyHorton]

There is, hopefully, significant overlap between the long term interests of PANOPTES and the short term needs of Huntsman, but arguably these are separate issues.

It sounds like what you're thinking of is a major rewrite of POCS to turn it into robust, adaptive, and deeply distributed system, something I think we would all agree should be on the roadmap. On the other hand what I had in mind when opening this Issue was the absolute minimum change required to enable cameras to be controlled by their own Raspberry Pi's (in our case Pi 3 B+'s). We should certainly try to make the latter a step along the path towards the former to avoid later duplication of effort, but that may not be practical.

It was a strong desire to limit the scope of the changes that lead me to start with Pyro in #547. It may not be the best solution long term, but it did offer the prospect of a workable solution with minimal changes to existing code. It makes it possible to address the Issue by adding additional optional functionality, rather than altering the existing architecture.

I see parallels here with the SBIG and FLI CCD cameras, and Birger Engineering and Optec focus controllers, which were added to POCS because Huntsman (and hopefully other projects, in the future) needed them.

[jamessynge]

I'm happy to hear that you're assuming for the minimum change. The PR's list of tasks done and in progress made it seem rather more significant than you describe here. I'll take a look at it.

On Mon, Aug 13, 2018, 11:41 AM Anthony Horton notifications@github.com wrote:

There is, hopefully, significant overlap between the long term interests of PANOPTES and the short term needs of Huntsman, but arguably these are separate issues.

It sounds like what you're thinking of is a major rewrite of PANOPTES to turn it into robust, adaptive, and deeply distributed system, something I think we would all agree should be on the roadmap. On the other hand what I had in mind when opening this Issue was the absolute minimum change required to enable cameras to be controlled by their own Raspberry Pi's (in our case Pi 3 B+'s). We should certainly try to make the latter a step along the path towards the former to avoid later duplication of effort, but that may not be practical.

It was a strong desire to limit the scope of the changes that lead me to start with Pyro in #547 https://github.com/panoptes/POCS/pull/547. It may not be the best solution long term, but it did offer the prospect of a workable solution with minimal changes to existing code. It makes it possible to address the Issue by adding additional optional functionality, rather than altering the existing architecture.

I see parallels here with the SBIG and FLI CCD cameras, and Birger Engineering and Optec focus controllers, which were added to POCS because Huntsman (and hopefully other projects, in the future) needed them.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/panoptes/POCS/issues/546#issuecomment-412563150, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYmU0jNtVhukT2kYi7RLxRMs_DqzU0gks5uQZ41gaJpZM4V6JtZ .

[wtgee]

Agree with @AnthonyHorton, I don't think this introduces any major complications and puts some of the camera line in work with the long term road-map of decoupling the hardware from the Observatory, which was agreed on a long time ago as any first step toward moving to a more decentralized system.

@jamessynge I sent you the info on Pyro4 a few weeks back and mentioned that was the most likely option that was being looked at. It shouldn't be incompatible with indilib/indigo. And, I suppose I should point out, this is making an individual camera work with a Pi, and with far less effort than an immediate switch to something like indilib, which would not at all be practical right now.

Also @jamessynge you should not be having issues with the scheduler (I believe your system is out of date) and the other bug with slewing was also fixed within a day of it happening. If you are still having issues lets work on those in their respective Issue.

[jamessynge]

Having reviewed the code, I can't say I'm that happy with the addition "as is". It seems to me that we will have two communication mechanisms: zeromq and pyro4. I'd feel better if we had clear API for communicating between components of the system, which includes:

• Scheduler / Planner
• Observatory Control (roughly the state machine for safely executing an observation)
• Sensors / Actuators
  • Weather / Environmental
  • Electrical / Mechanical (e.g. current, voltage or mount limit switches, and relays)
• Cameras / Focusers / Filters
• Mount
• Dome
• Data Storage (local and remote)
• Analysis (e.g. astrometry, which might be local, might be remote)
• Logging

Along with that, we'd need something like the ability to discover components, preferably dynamically to allow for devices being power-cycled or for software upgrades being applied. This probably means that having different levels of optionality: one of N cameras being missing is OK, while the weather station failing to respond is considered a safety hazard.

[AnthonyHorton]

@jamessynge Again, what you describe is well beyond the intended scope of this Issue and the corresponding Pull Request.

My goal here is to address a single, specific, well defined, and immediate need: the ability to run Camera (& Focuser) objects on individual computers, separate from the main control computer that hosts the Observatory object. Please evaluate the proposed changes in that context, rather than measuring them against the larger goals from the long term roadmap.

Despite being primarily motivated by an immediate need the Pull Request does move POCS towards the long term goals you discussed. It introduces, for the first time, automatic discovery of components. Only Camera + Focuser for now, and only at startup rather than dynamically, but I don't think that's bad for a single PR.

A standard API for communications between and discovery of all components of POCS would be great, but that's a much larger discussion that I think belongs in a different Issue (and a whole series of PRs).

[wtgee]

I think the concern would be that once the Pyro4 code goes into the main codebase it would be hard to ever remove. I tend to agree that I don't seeing it act as a wholesale replacement of a distributed system and therefore I too would be worried about supporting it long term.

I know @jamessynge is a fan of the INDI library and wants to push us in that direction and Pyro4 seems to be doing the same thing but in a divergent manner. I would prefer to see some kind of lightweight PanDaemon kind of solution that ultimately uses zeromq. However, it's my fault for not getting the conversation started on spec'ing this out before @AnthonyHorton had to start on some work.

Anthony, we could pull this into the huntsman-pocs repo without too much fanfare (I believe). If it matures as a solution that we like we can then PR it into POCS itself. Thoughts?

[AnthonyHorton]

Yep. Well, as I noted in my first comment over in the PR:

Huntsman needs this urgently. Thus I'm inclined to proceed in the meantime anyway, in order to get something working well enough that Huntsman can start using it. If review comes up with a better approach and none of this stuff ever gets merged that's fine, but we need something now.

So, I'm OK with this not making it into POCS if something better based on zeromq or INDI provides the functionality, though will naturally be disappointed given the amount of work I've had to put into it. In the absence of a better design I had to get on with it myself as best I can, which given the time constraints and my experience meant Pyro.

Shunting this over to huntsman-pocs works in as much as I don't think there would be any technical difficulties with doing so, and it solves the immediate problem for Huntsman. Really need to fix those private repo & Travis problems though, being unable to share the code or use CI makes huntsman-pocs a much less desirable home for my efforts than POCS.

[jamessynge]

@AnthonyHorton I should have been clearer: I'm not happy, but I understand your need. We often have to take on technical debt to add necessary features, i.e. degrade our beloved architectural cleanliness. So, I can tolerate adding this feature, as long as we recognize the need to not continue to make it harder to evolve.

@wtgee What I'd like to see is that we immediately start gathering input on the weaknesses of our current code base w.r.t. communication, coupling, robustness and extensibility, and then use that to guide our discussion regarding evolving the architecture of POCS.

Regarding Indilib, I'm not arguing that we should use it, but rather that we should seriously evaluate it. Lots of folks have clearly put in a lot of effort developing it, and we have so few resources available for POCS that we shouldn't ignore the potential benefits of adopting Indilib (or Indigo or Skynet).

I also want to investigate what frameworks exist that we could exploit for achieving greater robustness, extensibility and looser coupling. I've not yet found something that seems super strong in the Python realm, but will do some more research.

Does that sound reasonable?

[jamessynge]

Going rather off-topic...

Regarding Python, I don't love it as a choice for POCS simply because we find out about several classes of problems only at runtime (e.g. type mismatches and unknown symbols). Yet I recognize that Python is much more approachable for novice programmers than most compiled languages. This has led me to consider the benefits of POCS providing a framework written in a strongly typed language, but with extensibility with any language enabled via either a communication system like ZeroMQ, a protocol like HTTP (e.g. json RPCs) or even a simple unix pipe. This would allow schedulers and image analysis code to be in Python.

The advantage of an RPC like mechanism is that we don't have to do as much upfront design of our communications, and only need to focus on pair-wise contracts. OTOH, ZeroMQ, being more of a pub-sub system, seems better suited to a message-bus style. The latter can be very useful in that we don't need to know who the recipient of a message is, just what the format must be... which requires more planning.

[gnthibault]

My 2 cents here (amateur astronomer). I was trying to design a POCS-like software (for fun/ learning experience) for some month now. I found POCS while trying to look for a proper FSM support in python, and was really amazed by its features/the way it was written because it was very close to my ideal goal.

My new branch is now heavily inspired by POCS, and some of its files are just direct copies of it, please let me know if that is a problem/I should update some reference/licence files somewhere. Implementing an Indi version of the mount control class from POCS, is in my TODO list, although I haven't started yet.

Anyway, I first started with my own classes for providing services (astrometry, weather...) and relying on Indi for driving other devices like mount/camera or focuser.

Of course, I am nowhere near something usable now (I don't have time constraint, it is a personal project), but you can find example of pyindi usage and other services here: https://github.com/gnthibault/RemoteObservatory/tree/NewScopeSimulator

An example of an indi mount class in python: https://github.com/gnthibault/RemoteObservatory/blob/NewScopeSimulator/Mount/IndiMount.py

Example of Astrometry client (corrected with help from dustin Lang): https://github.com/gnthibault/RemoteObservatory/blob/NewScopeSimulator/Service/NovaAstrometryService.py

Sorry if that message disturbed the aforementioned issue tracking.

Regards

Thibault

[jamessynge]

Thibault, thanks for your input. After reading it I decided to post to our forum regarding the general topic of build/borrow/buy of this software. Perhaps you'd like to chime in?

[wtgee]

Closed by https://github.com/AstroHuntsman/huntsman-pocs/pull/80