search

openfaas / mqtt-connector

MQTT connector for OpenFaaS
MIT License
34 stars 10 forks source link

Question - is there multiple topic support? #5

Open flusflas opened 3 years ago

flusflas commented 3 years ago

My actions before raising this issue

Expected Behaviour

This connector cannot deal with more than one MQTT topic, as stated in https://github.com/openfaas/mqtt-connector/tree/master/chart/mqtt-connector and confirmed by my own tests with the connector. This seems normal as only one MQTT subscription is created. I'm wondering why this connector is limited to one single topic per instance.

Current Behaviour

Only one topic is allowed. If more than one topic are used (e.g. my-topic,my-second-topic), no errors are thrown but only the first topic seems to be used. Only messages published to the first topic my-topic will be received by the connector.

Possible Solution

I've added very simple multi-topic support using multiple subscriptions, but maybe I'm missing how the connector-sdk handles/splits the topic internally.

Your Environment

alexellis commented 3 years ago

Please do us a favour and use the issue template, this is required for everyone and there are no special cases.

https://raw.githubusercontent.com/openfaas/mqtt-connector/master/.github/ISSUE_TEMPLATE.md

Incidentally, it looks like multiple topics may already be supported - https://github.com/openfaas/mqtt-connector/blob/master/main.go#L70

You can see an example usage, if it's of interest to you here -> https://github.com/packet-labs/iot/tree/master/openfaas/mqtt-connector

Ping me here again when you've updated the description of your issue.

alexellis commented 3 years ago

/set title: Question - is there multiple topic support?

flusflas commented 3 years ago

Description updated. Based on my tests (and the connector documentation), only one topic is allowed as only one MQTT subscription is being created.

max-regan commented 3 years ago

I was also hoping for this functionality. I believe the code @alexellis referred to actually allows a function to have multiple "topic" annotations, not for the MQTT connector to subscribe to multiple MQTT topics.

I'm not currently in a position to test, however I believe that you may be able to have some success with using topic wildcards such as '#', which would subscribe to every topic.

Related source: https://github.com/eclipse/paho.mqtt.golang/blob/master/topic.go

Unfortunately I'm not in a position to test right now. Hopefully that helps!

flusflas commented 3 years ago

I've seen that in https://github.com/openfaas/mqtt-connector/blob/d6f8f43c072e7d3276b8ec36bad08900565816b8/main.go#L93-L95 a default publish handler is being used, so all messages in all topics are being received. As far as I can see, it makes both the topic list and the subscription in https://github.com/openfaas/mqtt-connector/blob/d6f8f43c072e7d3276b8ec36bad08900565816b8/main.go#L102 completely useless.

Wouldn't a MessageHandler callback on the subscription line make more sense? This will allow to listen only to the requested topics. Of course, only one topic per subscription will be allowed. I guess is in that scenario where my question about multi-topic support made sense.

aslanpour commented 2 years ago

I think multi-topic is supported by mqtt-connector. It just needs to be made a multi-topic handler by using '#' .

For instance, the connector can set a topic as '/function/#'. Function 'test1' can be annotated by topic='/function/test1'. Function 'test2' can be annotated by topic='/function/test2'. Function 'test3' can be annotated by topic='/function/test3'.

Then by publishing a message on the topic 'function/test1', the connector will be notified and will only invoke function 'test1'.

I tested this and worked.

I would just understand the openfaas components that an MQTT message visits in this trip.
My understanding is that the MQTT message is delivered to MQTT broker --> mqtt-connector --> openfaas gateway --> function and then results are sent back to mqtt-connector that I do not understand why and how we can catch the result from mqtt-connector. In case of async, the mqtt message is delivered to MQTT broker --> mqtt-connector --> nats --> queue-worker --> function -and results are sent back to mqtt-connector that I do not understand how we can set a callback here. If someone is familiar openfaas design, please correct me.

alexellis commented 2 years ago

@flusflas you didn't fill out the issue template and left off the "Steps to reproduce this issue" part

If you haven't tried this out, how do you know it doesn't work?

And before we get ahead of ourselves, are you talkigng about 1000 topcis, or 3?

If it's a low amount of topics, just install the connector N times per topic. That's a simple and easy solution.

@aslanpour please feel free to try out the scenario you listed above with a wildcard topic using #

aslanpour commented 2 years ago

@alexellis I just tried the scenario out and works for me. So, no need for a connector per topic. If you need a demonstration, I should be able to give more info in this thread.

alexellis commented 2 years ago

That'd be great. I can then add it to the README after that for everyone to benefit from.

flusflas commented 2 years ago

Of course you can use a topic with wildcards to address more than one topic, but it doesn't allow multiple topics if they don't share a common structure or you don't want to use wildcards. I think the problem was clear enough and that I provided detailed information on where the problem was.

Anyway, I chose to create a fork of this repository to add this and other features to the connectors, so feel free to close this issue if the current behavior is fine for you.

alexellis commented 2 years ago

When you're asking someone to do work for you (out of good will, voluntarily) and they say: "this isn't clear enough", replying with "it is clear enough" means that we have a bit of a communication breakdown.

You're perfectly welcome to fork this repository for your own needs. That's why it's licensed as MIT, however you cannot change the copyright notices and need to credit the source.

If you're using openfaas for production or in a proprietary solution, then you should consider becoming a sponsor. That is, if you would like to see us continue to maintain the project and make it available for you.

https://github.com/sponsors/openfaas

alexellis commented 2 years ago

/lock: off topic