Proposal to change Google pub/sub scaler

[IanMoroney]

Proposal

Now that it's possible to pull messages directly from pub/sub without acknowledging them, a more direct integration with Google pub/sub should be possible. https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions/pull

Use-Case

With the current num_undelivered_messages metric implementation, it takes 2-3 minutes between when a message is dropped on the queue and when the metric is updated. This is unacceptable in terms of lag for near real-time operations when keda is meant to scale based on a message appearing in the queue.

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

[JorTurFer]

Hello Are you sure that you can get the queue length directly? The link you sent pulls the messages from the queue directly

[IanMoroney]

By pulling the messages, you can count the queue length. Pulling without acking leaves the messages on the queue

[IanMoroney]

Thinking about it, that could present a problem for a lot of messages, so wouldn't necessarily scale well.

[rickbrouwer]

Yeah, if there are a significant number of messages on it, this can become an issue.

[JorTurFer]

I see multiple issues on that approach:

• Operation problem with large message count or huge messages (KEDA would need to pull X GB just to count the length)
• Accessing management as KEDA will read the messages and potentially their content

I agree with you in the point of current approach doesn't work for real time scenarios due to the consolidation lag in monitoring APIs, but we prefer to keep KEDA "agnostic" of the data plane to prevent potential data leaks. All the scalers work at control plane level

[IanMoroney]

In case they actually look at implementing something like this, i've formally requested it in the Google issue tracker for pub/sub. If you'd like to comment on it to see if it gets any traction: https://issuetracker.google.com/issues/372044259