jeffbass
commented
3 years ago Hi @sbkirby,
I’m glad you enjoyed my PyCon 2020 talk. My vision for the librarian has evolved over time. In 2018, I designed and wrote a first draft. I ran that first version for over a year (the text message exchange in the README.rst is a screenshot from the first draft of the librarian). I ended up throwing the first draft away and starting over.
In my first draft, the librarian was sending return messages to the imagehub and then to the imagenodes using imageZMQ REP messages. It didn’t work well. I learned (and continue to learn) from the redesign and rewrite. In my current librarian design, the “two-way arrow” is a one way arrow from the imagehubs to the librarian. All the communication from imagehubs to the librarian is done through the event log and image files written by each imagehub. The librarian "receives" this communication by reading those files and processing the images and event messages. As each imagehub updates these files, the librarian watches for and grabs any new images or event messages.
The way the librarian communicates with the imagenodes is also different than the diagram. All the images and event messages go one way from imagenodes to imagehubs. The imagehubs do nothing except rapidly store received images and event messages. The librarian communicates "imagenode commands" directly to imagenodes. When the librarian gets a request like “send 3 images from the water meter right now”, it just changes the imagenode.yaml file residing on the appropriate water meter imagenode and restarts the imagenode. The imagenode restarts and does what the new imagenode.yaml specifies (e.g., send 3 immediate images, or turn off a camera detector, etc.). The librarian only communicates with the imagenodes through their yaml files now. It made the implementation of communications much simpler. I should re-draw the design diagram to reflect this, but I haven’t yet. For starting and restarting the imagenodes, I am using the Linux systemd / systemctl system to launch and restart the imagenode.py program. An example of the imagenode.service file is in the imagenode GitHub repository. This is working well for me. Trying to add complex 2 way communication protocols to the imagenode and imagehub programs was difficult and fragile. In my current librarian design, each time I need to implement a new imagenode “control task”, I think of a way to implement that feature by adding a new option to the imagenode.yaml option list. I then implement the new option in the imagenode code. The librarian communication of commands to the imagenodes is a sequence of "ssh exec imagenode restart” calls sent directly to each specific imagenode. These ssh calls copy the specific imagenode.some_command.yaml to imagenode.yaml, then restart the imagenode. To change it back to the previous imagenode.yaml, it performs a second copy and restarts again. All the various imagenode.some_command.yaml files reside in the home directory on each imagenode, so “sending a command to an imagenode” looks like this (each command is issued via ssh pi@rpi to the specific imagenode that needs to execute a command):
cp imagenode.no-motion.yaml imagenode.yaml # the no-motion yaml file is one WITHOUT a motion detector option
sudo systemctl restart imagenode.service
# the above commands set the imagenode to “stop motion detection”
# the below commands set the imagenode to “start motion detection”
cp imagenode.motion.yaml imagenode.yaml # the motion yaml file is one WITH a motion detector
sudo systemctl restart imagenode.serviceThese command sequences are in yaml files in the librarian_data directory; they are read by the librarian and sent as ssh commands to the specific imagenode via Python’s subprocess.run() module. It is necessary to configure password-less SSH on each RPi using ssh-keygen. Each restart of the imagenode program on a RPi takes about 3 seconds; that is fast enough for my applications.
Here's what my current draft of librarian does in pseudo code:
# stuff done one time at program startup
Read a librarian YAML file into Settings (imagehub file locations, gmail credentials, backup schedule, etc.)
Instantiate a Librarian using Settings:
Instantiate all CommChannel's (includes CLI, Gmail, etc.)
Do a system self check and log the result
Start threads and subprocesses:
Start a watching thread for each comm channel (e.g., gmail_watcher.py)
Start a watching thread that processes new data as it is saved by imagehubs
Start a scheduling thread that runs scheduled tasks like backups
Start a HealthMonitor subprocess that watches the librarian, imagehubs and imagenodes for issues
# The main Librarian forever loop:
Response Loop forever:
Receive a QueryRequest (from in_q which is loaded by an inbound CommChannel thread)
Optionally, take an action (e.g., set barn_cam to no_motion)
Compose a Reply (using data that was processed from the imagehub files)
Send Reply (put into out_q which will be sent in an outbound CommChannel thread)The librarian communications, image skills, data skills and other modules are located in a "helpers" directory. Here are the main Librarian Classes::
Class Settings (filled once by reading from the YAML file)
Class HealthMonitor (methods that monitor system and network health; does backups)
Class Librarian (instantiated once using Settings)
Class CommChannel (instantiated once per comm channel like Gmail or CLI)
Class Gmail (instantiated once; fetches and sends messages via Gmail API)
Class Chatbot (instantiated once; receives queries and composes replies)
Class HubData (instantiated once; provides HubData, e.g. events, to ChatBot)
Class Schedule (instantiated once; starts scheduled tasks and methods)
Class Imagehub (instantiated once per imagehub; reads imagehub saved data)
Class Imagenode (instantiated once for each imagnode)
Class Location (will instantiate as many locations as imagenodes)
Class ImagenodeCommand (instantiated each time a specific command is sent)
Class ObjectDetector (instantiated for each image and imagenode type)
Class WaterMeterReader (instantiated once; reads water meter digits from water meter image)
Class Person (can instantiate one or more Persons to answer queries from)Separate helper programs interact with the librarian; here is one example; there are about a dozen of these helper programs so far:
CLI_chat.py: A CLI program that sends queries to Librarian and prints responses to standard output (typically the terminal). The CLI_chat.py program is located in the communications module in the "helpers" directory.
My goal is to have my current version of the librarian in a GitHub repository by December or January so that I can propose a talk about it for PyCon 2021.
Thanks for your interest. Good luck with your own Distributed Computer Vision system! I look forward to seeing anything you might choose to share on GitHub. Jeff
P.S. I pushed a new version of imagenode to GitHub recently; it added an option for threaded image sending capability. It speeds up image sending by about 30%.
sbkirby
Jeff, I enjoyed your PyCon 2020 Talk about the Yin-Yang-Ranch. Thank you for all the hard work developing imagenode, imagehub and imagezmq. I've been spending some time building a Distributed Computer Vision system at home using your tools, and I'm at the point where I would like to learn more about how you integrated (or plan to integrate) the librarian into the system. You put a huge two-way arrow between the imagehub and the librarian on your illustration. Would you explain the operation you had in mind for that portion of your overall design? Thanks