search

firebase / extensions

Source code for official Firebase extensions
https://firebase.google.com/products/extensions
Apache License 2.0
887 stars 376 forks source link

[firestore-counter] no available instance log in firebase distributed counter #889

Open haris-wilson opened 2 years ago

haris-wilson commented 2 years ago

Bug in Distributed counters extension. The firebase log shows this message :

The request was aborted because there was no available instance. Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability.

Screenshot (106)

I am concerned about the extra billing caused by the errors.

dackers86 commented 2 years ago

Hi @haris-wilson

Although a number of updates are currently ready for release on this extension https://github.com/firebase/extensions/pull/817.

This seems like a potential new issue i'll can add this to the project board for further investigation.

danReynolds commented 2 years ago

Also concerned about the extra billing here. What's going on?

i14h commented 2 years ago

@danReynolds what is the extra billing concern? Can you explain a bit more?

haris-wilson commented 2 years ago

@i14h there is a huge log list being generated for my project which if left unwatched causes increase in billing.

cabljac commented 1 year ago

Looking into this issue. Can anyone confirm that this is still happening? If so I'd like to investigate.

From what I understand, if there is no available instance, then we are generating an error log on each onWrite trigger, and this could lead to unnecessary billing. (Correct me if i'm wrong)

shaibt commented 1 year ago

hey @cabljac,

I can confirm that in our deployment of the Distributed Counter extension, we're seeing multiple 429 warnings (in the 10s) daily for The request was aborted because there was no available instance. Beyond the increased logging, does this warning mean the function was aborted completely and won't be retried at all? if so, we're missing on a lot of realtime updates to our counters.

habibKadiri commented 1 year ago

Any updates on the issue at hand? We've just added this extension and began receiving these errors and warnings.

Asuveroz commented 10 months ago

In the last 12 hours, my project has logged over 32K warnings (POST 429) with the message: The request was aborted because there was no available instance. Should I manually increase the instance count from 1 or leave it like that? Also also noticed some messages on the controllerCore: Query to find shards to aggregate failed. Error: 10 ABORTED: Aborted due to cross-transaction contention. This occurs when multiple transactions attempt to access the same data, requiring Firestore to abort at least one in order to enforce serializability. I've been facing this issue for months now.

cabljac commented 10 months ago

Resuming investigation of this now. I've seen in other contexts The request was aborted because there was no available instance when there aren't enough cloud function instances for the events coming through. I'll attempt to repro first

EDIT: reproduced this, continuing to investigate

christianlacerda commented 2 weeks ago

I am seeing the same issue. Has anyone fixed it?