Several users observed a "syncinc-forever" bug that appeared to be related to the event stream panicking. One found the following error message:
Failed to deserialize EQS response: timed out.
My wallet (which was open over the weekend) also panicked, with:
Failed to deserialize EQS response: ResponseBodyError(None): unknown error.
Restarting my wallet, it was able to catch up to the latest state, so I believe what happened was some kind of transient issue with the EQS or load balancer, where we received a response with status code 200 whose body was not a list of events. Currently the wallet assumes that either the response has an error status or else it's body will deserialize correctly. Upon restarting the wallet, the transient issue with the EQS had resolved, and the wallet was able to replay events normally.
The solution here is to just make the wallet more resilient to this possibility, so a response body that fails to deserialize is treated the same as an error response (although possibly with louder logging, as this is more of an exceptional case).
Several users observed a "syncinc-forever" bug that appeared to be related to the event stream panicking. One found the following error message:
Failed to deserialize EQS response: timed out.My wallet (which was open over the weekend) also panicked, with:
Failed to deserialize EQS response: ResponseBodyError(None): unknown error.Restarting my wallet, it was able to catch up to the latest state, so I believe what happened was some kind of transient issue with the EQS or load balancer, where we received a response with status code 200 whose body was not a list of events. Currently the wallet assumes that either the response has an error status or else it's body will deserialize correctly. Upon restarting the wallet, the transient issue with the EQS had resolved, and the wallet was able to replay events normally.
The solution here is to just make the wallet more resilient to this possibility, so a response body that fails to deserialize is treated the same as an error response (although possibly with louder logging, as this is more of an exceptional case).