Rest+ stops changing state base on homekit commands

[HammerAce42]

I have found that I am often getting into a state where the Rest+ stops responding to any commands from homekit (on/off, brightness, color change).

Looking at the logs, I see the following error when this occurs:

[3/5/2020, 1:08:32 AM] [HatchBabyRest] AWS Iot Error
[3/5/2020, 1:08:32 AM] [HatchBabyRest] Error: unexpected server response (403)
    at ClientRequest.<anonymous> (/homebridge/node_modules/homebridge-hatch-baby-rest/node_modules/ws/lib/WebSocket.js:653:21)
    at ClientRequest.emit (events.js:323:22)
    at ClientRequest.EventEmitter.emit (domain.js:482:12)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:603:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)
    at TLSSocket.socketOnData (_http_client.js:476:22)
    at TLSSocket.emit (events.js:311:20)
    at TLSSocket.EventEmitter.emit (domain.js:482:12)
    at addChunk (_stream_readable.js:294:12)
    at readableAddChunk (_stream_readable.js:275:11)
    at TLSSocket.Readable.push (_stream_readable.js:209:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:186:23) {
  type: 'error',
  target: WebSocket {
    _events: [Object: null prototype] {},
    _eventsCount: 0,
    _maxListeners: undefined,
    readyState: 3,
    bytesReceived: 0,
    extensions: null,
    protocol: '',
    _binaryType: 'arraybuffer',
    _finalize: [Function: bound finalize],
    _closeFrameReceived: false,
    _closeFrameSent: false,
    _closeMessage: '',
    _closeTimer: null,
    _finalized: true,
    _closeCode: 1006,
    _receiver: null,
    _sender: null,
    _socket: null,
    _ultron: null,
    protocolVersion: 13,
    _isServer: false,
    url: [*redacted*]
    _req: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [TLSSocket],
      connection: [TLSSocket],
      _header: [*redacted*]
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      insecureHTTPParser: undefined,
      path: [*redacted*]
      _ended: false,
      res: [IncomingMessage],
      aborted: true,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    [Symbol(kCapture)]: false
  }
}

It appears that the only way to clear the issue when it occurs is to restart homebridge.

[dgreif]

@HammerAce42 I wonder if there is a lifespan on the auth tokens that I'm not accounting for yet. I restart my homebridge instance every couple days due to updates for other plugins, so I haven't run into this issue yet. Do you have a rough estimate on how long it takes for it to happen?

[HammerAce42]

It was certainly less than 48 hours since I last restarted homebridge to the point this issue occurred. However, I did have to reboot my router/disrupt internet connectivity a few times in between which may have caused the Rest+ to get a new auth token. Perhaps a new auth token should be requested as an automatic step anytime an error like the one I posted above occurs?

[HammerAce42]

The issue repeated when I intentionally took down my internet to test this, so it's almost certainly caused by a disruption in connectivity.

[dgreif]

@HammerAce42 thanks for all of the info! I think I have enough to go off of for working on a fix, but no promises on when I will be able to get it released

[HammerAce42]

@dgreif, I am having this issue regularly despite my internet/wifi being consistently online (as far as I can tell). My theory around it being due to connectivity resets is probably incorrect. Just wanted to let you know in case that affects the fix that you come up with.

Thanks!

[HammerAce42]

Were there any changes in v2 that might have helped with this? I haven't seen the issue arise since I got 2.0.1 working yesterday, and am hoping that's not a coincidence :)

[dgreif]

I did update all dependencies in v2, which includes the library used to maintain that connection. It's possible that they fixed something in that library. Keep me posted!

[HammerAce42]

Unfortunately the issue is still present. I am seeing the same error as included in the earlier logs.

[HammerAce42]

Friendly nudge on this! :)

[dgreif]

@HammerAce42 sorry for the neglect, I've been spending all my time on homebridge-ring and haven't had a chance to look into it. Are you still having issues? (Assuming the answer is yes lol). I haven't seen this at all, but I think my homebridge resets at lease once every 24 hours because of another plugin that I'm using that has uncaught errors.

[HammerAce42]

No worries - thanks for your efforts on the ring plugin - I benefit from that as well :)

This issue still occurs regularly for me - I haven't figured out exactly what the expiration time period is. I end up having to manually restart Homebridge whenever I see it.

[dgreif]

Sounds good, thanks for the info. I think I can detect when it errors, just gonna take some refactoring to get it to reconnect. Will get to it once Ring stabilizes 😄

[dgreif]

@HammerAce42 Just released v2.0.3 which adds logic to open a new MQTT client with aws if the login credentials expire. Give it a try for a few days and let me know how it goes!

[HammerAce42]

@dgreif Thanks - have updated and will post here if I see the issue again in the next few days.

[HammerAce42]

This issue hasn't reoccurred since I updated to the release with the fix. Closing it out.

Thanks again for getting this sorted!