search

absmach / magistrala

Industrial IoT Messaging and Device Management Platform
https://www.abstractmachines.fr/magistrala.html
Apache License 2.0
2.43k stars 664 forks source link

Writers nats msgHandler needs refactor #1068

Open mteodor opened 4 years ago

mteodor commented 4 years ago

Currently Writer's NATS msgHandler() is consuming NATS messages synchronosly with delivering message. It should be changed so that it follows pattern of passing things into a queue and then having a number of workers It can be automated with by using nc.ChanSubscribe or the SubscribeSync/NextMsg pattern

drasko commented 4 years ago

@mteodor can you please give links to nc.ChanSubscribe and SubscribeSync/NextMsg and explain better the concept and what do you want to achieve here.

We are not using Go channels here, and I am not sure that this is relevant or necessary for our use-case

mteodor commented 4 years ago

If you have a large number of large messages and writing takes a long time you may lose messages. With ChanSubscribe you will create a buffer for messages and process messages at pace.

https://github.com/mainflux/export/blob/ae3af57b13d7ce917f79a5e23a86974d4d9ac901/pkg/export/service.go#L184

drasko commented 4 years ago

These functions are using Go buffered channels, right?

This might be possible, because our Writers are all written in Go.

However, I would like here that we consider using NATS 2.0 and researchtheir QoS, i.e. queue and streaning that NATS 2.0 brings.