MQTT - based scaler

[arschles]

I'd like to have a scaler listen to an MQTT source and scale up/down based on incoming messages. This would be useful for Kubernetes clusters that run on the edge (for example, in IoT scenarios and in scenarios where k8s is running on underpowered machines)

• Scaler Source: A scaler to support scaling applications based on incoming MQTT messages. Not intended to support bidirectional communications
• How do you want to scale: Based on volume of messages on a particular topic
• Authentication: Two options are username/password auth and advanced (x.509) auth. username/password auth should be sufficient for now, given that you can already place credentials in a secret and pass them into a ScaledObject

[arschles]

I would be happy to get started on this.

[andschneider]

I'd be interested in helping out as well!

[zroubalik]

I like the idea. It's up to you guys, who wants to work on this! :) Thanks!

[iennae]

@zroubalik what does the "needs discussion" label mean? It's ok to start implementing right?

[andschneider]

I started looking into making the scaler and was trying to figure out what to use as the metric value.

Since MQTT brokers don't have the concept of a queue, using queue length or similar wouldn't work. I was then thinking of using number of messages received over a certain period of time. However, I'm not sure if time based scalers are possible. @arschles Is this what you meant with "volume of messages"?

[arschles]

I didn't have anything specific in mind regarding "volume of messages" - just an abstract idea. I think that number of messages received over a period of time could work. another idea is to maintain a rolling average of messages received over a window. according to you, @andschneider , you'd need to have KEDA be a subscriber to the queue - not sure if that's ok to do. @zroubalik thoughts on that part?

[zroubalik]

@zroubalik what does the "needs discussion" label mean? It's ok to start implementing right?

that means that this issue needs some clarification/specification/proposal on how it will be implemented, basically the stuff that you can see is happening.

[zroubalik]

@arschles so your idea is, that KEDA will subscribe to the topic and will do the calculation (messages received over a window etc)? I am not an expert on MQTT, I wonder, are there any metrics collected by the MQTT Broker itself?

[andschneider]

I'm by no means an MQTT expert either so take this all with a grain of salt, but I did a little more research and it seems that some brokers do expose metrics. As it's not part of the MQTT spec it's up to the broker's implementation for what they are though. For example, mosquitto and HiveMQ seem to be popular choices and they expose different metrics.

To keep the scaler as general as possible my thought was to stick to something in the spec. There's a thing called retained messages which is flag set on a published message that basically tells the broker to always store one (and only one) message for a topic, until it gets deleted. Essentially it could be used like a status message and if present the scaler would spin something up.

Obviously a little different than using some sort of messages in a time span/window approach - but it gets around having the scaler subscribed to a topic. @arschles @zroubalik what do you guys think?

[arschles]

@andschneider would you be up for getting this started? A lot of my time is taken getting the http-add-on started

[andschneider]

@arschles Yep, if everyone is cool with the direction I'll have some time towards the end of this week to get it going.

[andschneider]

@arschles Apologies for the delay - I finally started working on it this weekend.

Here's a repo with an example setup for running a MQTT broker and pub/sub components. The basic scaler seems to be working (its on the mqtt-scaler branch of my Keda fork). Main things left to do are auth, higher QoS messages, and general robustness.

[tomkerkhove]

Great to hear, thanks!

[andschneider]

Quick update - with holidays and whatnot I haven't been able to spend much time on this. Next week is finally looking all clear though and I should be able to make some progress!

[tomkerkhove]

That's great, thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[arschles]

@zroubalik it doesn't seem like there's much support behind the MQTT-based scaler. should we close this or let the stale bot do it in due time?

[zroubalik]

@arschles let's keep this open as it is a valid ask, maybe someone else will pick it up in the future 🤞

[arschles]

Sounds good!

[vainkop]

MQTT scaler would be definitely good for AWS iot core processing https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html