search

AstroHuntsman / huntsman-pocs

POCS on the Huntsman Telescope
MIT License
7 stars 8 forks source link

Pyro objects do not unregister from nameserver #251

Open danjampro opened 3 years ago

danjampro commented 3 years ago

If a pyro device is terminated (or crashes), it should unregister itself on the name server. Additionally, when starting a new pyro service, we should unregister any pyro object with the same service name to prevent registration errors.

This is particularly important for getting the restart on reboot working properly.

AnthonyHorton commented 3 years ago

They used to unregister themselves, it was in the camera server (now service) __del__() method. It's impossible to guarantee that gets run though, as it relies on Python being allowed to clean up properly. If the OoM killer just splats the Python process or the Pi hard resets no Python code is going to be able to ensure the remote object gets unregistered.

Because of that we also set the name server to automatically unregister any remote objects that it was unable to contact for a set timeout time, in order to clean up in case a camera server died without managing to unregister itself.

Have these features been translated to the new Pyro5 setup?

I like the idea of checking for and removing duplicates, though.

lspitler commented 3 years ago

I thought we'd attempt to reboot a pi fast enough so that the name server doesn't know it has gone offline?

lspitler commented 3 years ago

Relevant to: https://github.com/AstroHuntsman/huntsman-pocs/pull/243

lspitler commented 3 years ago

@AnthonyHorton would this work:


                    if self.config['pyro']['reboot_on_failure']:
                        self.logger.debug(f"Rebooting computer hosting camera {self._camera}")
                        self._camera.__del__()
                        time.sleep(5)
                        subprocess_args = ["sudo", "reboot", "now"]

in: https://github.com/AstroHuntsman/huntsman-pocs/blob/d4c6aa13a9697e847eceb1707266aa2f70a210fa/src/huntsman/pocs/camera/pyro/service.py ?