substratusai / lingo

Lightweight ML model proxy and autoscaler for kubernetes
https://www.substratus.ai
Apache License 2.0
96 stars 6 forks source link

Messenger: Panic panic: Ack/Nack called twice on #93

Closed samos123 closed 3 months ago

samos123 commented 3 months ago

There seems an unhandled error which causes lingo to crash and then the pod will get started, directly causing in termination of all pods because lingo will scale down from e.g. 100 replicas directly back to 1 after a restart.

This is the error seen in the logs before the failure:

panic: Ack/Nack called twice on message: &{LoggableID:10628088997508317 Body:[...