ecadlabs / signatory

Signatory - A Tezos Remote Signer for signing block-chain operations with private keys using YubiHSM, AWS, GCP, Ledger's or Azure Key Vault
https://signatory.io
Apache License 2.0
61 stars 18 forks source link

Rare occurence of the error "AsymmetricSign: context deadline exceeded" with GCP CloudKMS #497

Open Mayeu opened 2 months ago

Mayeu commented 2 months ago

Hello,

We are encountering a rare occurence of a "AsymmetricSign: context deadline exceeded" error using GCP CloudKMS. Rare as in, during the past year it only happened 4 times on our mainnet baker.

The full error (I replaced our project name with XXXXX):

Error signing request: (CloudKMS/projects/XXXXXXXXX/locations/global/keyRings/baking) AsymmetricSign: context deadline exceeded

I'm posting this here mostly in case this could be an issue for baker with a much bigger stack than ours (our address: tz3YJYNkKUktLkJXoWLyo3r6dTwfwGASVzUx)

It doesn't seem to be related to any activity on the VM, the CPU of this machine never goes above 10%, RAM is always lower than 20%, and disk activity is basically null.

We are running Signatory on a e2-highcpu-2 instance, as the only service there.