Fix problem with subscribing to internal Mainflux nats subject

[michaelkargl]

Hey,

I have been following along the mainflux/export instructions. The freshly built exporter runs, but can't seem to get any messages.

The exports taken were:

## ~/hello-mainflux/export/configs/config.toml
File = "../configs/config.toml"

[exp]
  cache_db = "0"
  cache_pass = ""
  cache_url = "localhost:6379"
  log_level = "debug"
  nats = "nats://127.0.0.1:4222"
  port = "8170"

[mqtt]
  ca_path = "ca.crt"
  cert_path = "thing.cert"
  host = "tcp://192.168.34.95:1883"
  mtls = false
  password = "8fee2626-3872-41d6-a727-14602cb25fa7"
  priv_key_path = "thing.key"
  qos = 0
  retain = false
  skip_tls_ver = true
  username = "8f185dfd-ff9d-4080-9186-f0a94f9b2afa"

[[routes]]
  mqtt_topic = "channels/a5905e1e-b1fe-409e-aee5-b272d22227d2/messages"
  nats_topic = '>'
  subtopic = ""
  type = ""
  workers = 0

#> ~/hello-mainflux/export/build
export MF_EXPORT_CONF_PATH="~/hello-mainflux/export/configs/config.toml"
./mainflux-export

Running this results in a valid connection to the 'mainflux-hub' server, but the 'gateway' is still waiting for messages:

root@edgex-gateway:~/hello-mainflux/export/build# ./mainflux-export
Configuration loaded from file ~/hello-mainflux/export/configs/config.toml
{"level":"debug","message":"Client export-8f185dfd-ff9d-4080-9186-f0a94f9b2afa connected","ts":"2020-07-28T12:29:59.169161549Z"}
{"level":"error","message":"Failed to create stream group: BUSYGROUP Consumer Group name already exists","ts":"2020-07-28T12:29:59.172656758Z"}
{"level":"info","message":"Stream group * created ","ts":"2020-07-28T12:29:59.174139062Z"}
{"level":"info","message":"Export service started, exposed port :8170","ts":"2020-07-28T12:29:59.175183665Z"}
{"level":"info","message":"Republish, waiting for stream data","ts":"2020-07-28T12:29:59.175808767Z"}

Changing the nats_topic to '.>' or '*' resulted in an error No routes have been configured

The Setup is two servers on different hosts with direct connection without firewall. The docker-compose file from mainflux/ui was used to setup the servers.

• Mainflux-Gateway: 192.168.34.95

  thing_id='4f570e52-8e25-49ed-81d8-221109126ddc'
  thing_key='15066dc1-9d8d-4094-9d7d-5819214c805c'
  topic='channels/7ff9d68a-f9d3-4ded-97a3-e8f388c8ad4a/messages'
  mainflux_broker='192.168.34.96'
  mainflux_broker_port=1883

• Mainflux-Hub: 192.168.34.95

  thing_id='8f185dfd-ff9d-4080-9186-f0a94f9b2afa'
  thing_key='8fee2626-3872-41d6-a727-14602cb25fa7'
  topic='channels/a5905e1e-b1fe-409e-aee5-b272d22227d2/messages'
  mainflux_broker='192.168.34.95'
  mainflux_broker_port=1883

For testing purposes a bash script is sending readings via

mosquitto_pub -u "$thing_id" \
                    -P "$thing_key" \
                    -t "$topic" \
                    -h "$mainflux_broker" \
                    -p "$mainflux_broker_port" \
                    -m "$message"

to the the Mainflux Gateway MQTT server. The readings are logged and can be visualized via Grafana. As instructed I opened up the port for the NATS servers 6222/tcp, 0.0.0.0:4222->4222/tcp, 8222/tcp and used an existing redis instance as cache es-redis

[mteodor]

To pick up messages from local mainflux instance you need to make nats_topic= "channels"

[michaelkargl]

Amazing! Your fixes worked. It sends data now. Unfortunately it seems to receive the wrong messages. Is there a special subtopic maybe?

Export Service:

{"level":"debug","message":"Published to: channels.7ff9d68a-f9d3-4ded-97a3-e8f388c8ad4a, payload: \n$7ff9d68a-f9d3-4ded-97a3-e8f388c8ad4a\u001a$4f570e52-8","ts":"2020-07-28T19:20:03.437240103Z"}

Mainflux:

mainflux-influxdb-writer | {"level":"warn","message":"Failed to handle Mainflux message: failed to decode s
enml : invalid character '$' looking for beginning of value","ts":"2020-07-28T19:19:59.38070581Z"}

Used messages from one of the examples:

[{"bn":"name","bu":"A", "n": "voltage", "u": "V", "v": 1755 }, { "n": "current", "t": -2, "v": 2888 },{ "n": "current", "t": 1, "v": 19431 }, { "n": "current", "t": 0, "v": 28065 }]

[drasko]

@mteodor is this documented in the official documentation? If not, please add it.

[mteodor]

@michaelkargl I've updated PR and this will fix problem with messages not getting saved however you have to take into account some changes MF_EXPORT_CONFIG_FILE replaces MF_EXPORT_CONF_PATH

and you need to specify route type

  nats_topic = "channels"
  subtopic = ""
  type = "mfx"

@drasko sure, docs will be updated

[michaelkargl]

With the new changes and configuration, the export succeeded! Big thanks to all of you!