Closed ypujante closed 10 years ago
What would be helpful is the following behavior on the console:
Agents can go bad every once in a while due to different reasons – in software development bugs happen :) But if console handles it gracefully then very little harm is done. Thanks again for your help!
I agree. Note that the fix will go in the next version of glu (5.5.x).
I was able to reproduce the issue on my machine. I do not know yet what the problem is but investigating...
Fixed in 4.7.3 and 5.5.1
Thank you! And thanks for patching the 4.x line.
@ykorabelnikov you are welcome.
My understanding of the bug makes me believe that it happened because during the upgrade:
1) the agents stops and restarts. 2) When it restarts it needs to re-instantiate the glu script. 3) Prior to 4.6.2, the glu script was not stored locally and was being fetched from its original location (as defined in the glu model). 4) If the original location is not accessible, then glu cannot re-instantiate the glu script and simply ignores this entry (the agent itself is fine). 5) after booting, the agent synchronizes the filesystem with ZooKeeper (Syncing filesystem <=> ZooKeeper message in the agent log) 6) this step (was) blindly loading all the states from the filesystem (which are java serialized objects) and storing them in ZooKeeper as json object 7) the issue is that in 4.7.1 the format of the file has changed and so because of 4) and 6) you end up with the wrong format in ZooKeeper for those states that were ignored in 4) 8) the console then receives this invalid data and then fails
What I did to fix the issue: 1) during the boot process, the agent will upgrade old format to new format 2) if a state cannot be restored, it is moved to a separate location and a "dummy" InvalidStateScript is instantiated so that it will appear in the console with the proper stack trace so that you can identify what the problem is 3) on the console side, it no longer fails if one entry cannot be read but will also instantiate a dummy one so that it is not silent
Technically the console should not see this problem because of the fix in the agent, but if you use the new console with an old agent, then at least the console will "survive" and continue to be operable.
Both 4.7.3 and 5.5.1 have those fixes.
Follow up from this thread: http://glu.977617.n3.nabble.com/Urgent-Need-help-td4026195.html