Shutter commands are sometimes not acknowledged

Reported by @rpineau

The find home now works very well at all speed. But .. There might be a new issue with this firmware. As expected we can get status message between a command and its response but usually the response comes back fairly quickly after. With the new firmware I see huge timeout between the command, status messages and the actual response :

[Sat Oct 26 16:37:08 2019] [CNexDomeV3::domeCommand] sending : @PRS

[Sat Oct 26 16:37:08 2019] [CNexDomeV3::domeCommand] response : 'XB->Online'
[Sat Oct 26 16:37:08 2019] [CNexDomeV3::domeCommand] XBee status : 'XB->Online'
[Sat Oct 26 16:37:10 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:12 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
^X[Sat Oct 26 16:37:14 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:14 2019] [CNexDomeV3::domeCommand] response : ':BV823'
[Sat Oct 26 16:37:14 2019] [CNexDomeV3::domeCommand] m_dShutterVolts : 12.07
[Sat Oct 26 16:37:16 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:18 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:18 2019] [CNexDomeV3::domeCommand] response : 'XB->Online'
[Sat Oct 26 16:37:18 2019] [CNexDomeV3::domeCommand] XBee status : 'XB->Online'
[Sat Oct 26 16:37:20 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:22 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:24 2019] CNexDomeV3::readResponse Timeout while waiting for response from controller
[Sat Oct 26 16:37:24 2019] [CNexDomeV3::getDomeEl] ERROR = PRR0

As you can see the @PRS command is sent at 16:37:08, then we get 2 "event notifications" (XB status and shutter voltage) at 16:37:08 and 16:37:14 and then another XB status at 16:37:18 and a response arrives at 16:37:24 , that's 16 seconds after sending the command, and it's not the proper responser (hence the ERROR label). So there is something weird going on when requesting shutter states. This is not happening often but I saw it once on connect and another time (above)

I see more response that do not match the command send :

[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] sending : @SWR

[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] response : 'STOP'
[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] response : 'Hstop 4'
[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] sending : @SWS

[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] response : ':SER,0,1,12347,0,30'  <==== This is not a proper response to @SWS
[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] sending : @PRR

[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] response : ':SWR'  <==== This is not a proper response to @PRR
[Sat Oct 26 16:37:24 2019] [CNexDomeV3::getDomeAz] szResp = SWR
[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] sending : @PRR

[Sat Oct 26 16:37:24 2019] [CNexDomeV3::domeCommand] response : ':PRR0'

Originally posted by @rpineau in https://github.com/nexdome/Firmware/pull/11#issuecomment-546649385

@rpineau later commented:

I agree that this is potentially due to a XBee response being lost as I only see these on shutter commands. On my test rig the 2 XBee are about 15cm from each other so distance here is not the issues. Between the 2 "XB->Online" status there are 10 seconds during which we also get a battery update. So could it also be that the Shutter XBee didn't receive the command and is therefore not responding to it.

nexdome / Firmware

Shutter commands are sometimes not acknowledged #13