buildbarn / bb-storage

Storage daemon, capable of storing data for the Remote Execution protocol
Apache License 2.0
142 stars 91 forks source link

Filesystem errors in bb-storage are recorded as "Unknown" in prometheus #163

Open lijunsong opened 1 year ago

lijunsong commented 1 year ago

When the underlying storage is full (due to a bad configuration), bb-storage throws the following errors back to worker.

 "status":{"code":2, "message":"Failed to store cached action result: Shard 1: Backend A: no space left on device"}}

However, from prometheus, the storage returns "Unknown" grpc code. Same for frontend write call (see the screenshot below).

screen_shot_2023-02-21_at_10 56 44_am