kostya
commented
4 years ago Looks like really rare concurrent situation, when in parallel application was deleted and in that time triggered depend_on on another. I would think how to avoid this.
Open rgaufman opened 4 years ago
kostya
commented
4 years ago Looks like really rare concurrent situation, when in parallel application was deleted and in that time triggered depend_on on another. I would think how to avoid this.
rgaufman
commented
4 years ago Thank you, indeed we recently switched to having a separate application per camera, before we used to have a single application for all cameras but it meant needing to restart everything when we made changes.
rgaufman
commented
4 years ago Any update? :)
kostya
commented
4 years ago i not see how to reproduce it and how this can happen, i can only imagine this is celluloid bug. Is it happen again?
rgaufman
commented
4 years ago Yes, it happens every few days on one of the 900 or so servers we have :( - So much so, I added a separate process that just tails the eye logs every minute looking for "Exception: undefined method `groups' for nil:NilClass" and restarts eye if it finds this line. It only happens when stopping and starting eye "applications".
kostya
commented
4 years ago Can you show how config look like schematically?
kostya
commented
4 years ago i pushed 0.10.1.pre with simple not nil check, maybe it helps, still not understand how this can happen. can you try it?
rgaufman
commented
4 years ago The config is a big complex, we use eye to start/stop all the processes required to record an IP camera, this includes proxy process, live view, detector, etc.
Before, we used to just stop eye completely, then start eye again. This worked, but meant if someone edited the settings of 1 camera, all cameras had to be restarted.
We changed the implementation to report a list of camera IDs that changed and then we run
system("bundle exec eye delete camera_#{id}")Once all the camera IDs were stopped and deleted, we run:
bundle exec eye load ../recorder.eyeWhich ensures all cameras are started, internally it gets a list of IDs using a simple get request to the rails app running locally.
Maybe this is the wrong way of doing it?
I'll try with 0.10.1.pre, thank you!
kostya
commented
4 years ago Your usage is quite strange, usually not need to call delete, only load command with updated config. It should delete and stop deleted from configs processes (if process have stop_on_delete true). And then you call restart for required processes.
rgaufman
commented
4 years ago Reason we need to delete is sometimes the camera configuration changes, so it's not just ensuring all cameras are loaded, but sometimes we need to delete the camera and then load again to make sure that camera is loaded back. We have stop_on_delete true already :)
One of my servers was misbehaving, so I sshed into it, checked the logs and found the following:
Any ideas what could be causing this and how to prevent in the future?