Filter messages sent from the edge

[marijadstankovic]

I'm trying to filter messages that are sent from edge to cloud. I want to do achieve what is described here but it's not working. export config file

File = "./export-config.toml"

[exp] cache_db = "0" cache_pass = "" cache_url = "localhost:6379" log_level = "debug" nats = "nats://127.0.0.1:4222" port = "8170"

[mqtt] ca_path = "ca.crt" client_cert = "" client_cert_key = "" client_cert_path = "thing.crt" client_priv_key_path = "thing.key" host = "tcp://192.168.0.100:1883" mtls = false password = "dcc10567-b1d8-424d-a7b3-c5e70cffe5bc" qos = 0 retain = false skip_tls_ver = true username = "333b6ee8-c3e5-4e61-a090-19cef0030146"

[[routes]] mqtt_topic = "channels/dd5b8d8b-086b-4d7b-9c9e-dc5c5835d492/messages" nats_topic = "export" subtopic = "" type = "default" workers = 10

[[routes]] mqtt_topic = "channels/dd5b8d8b-086b-4d7b-9c9e-dc5c5835d492/messages" nats_topic = "channels.f26f6357-1a58-4f63-9198-c68628df173e" subtopic = "my_subj" type = "mfx" workers = 10

publishing message mosquitto_pub -d -u f6b7e529-5adb-41ff-8666-d4b2fad6d769 -P e687d1fe-a779-414b-9d5d-1b61d87d9f5d -t channels/f26f6357-1a58-4f63-9198-c68628df173e/messages -h localhost -p 1883 -m '[{"bn":"1:", "n":"exec", "vs":"ls, -l"}]'

when I remove .f26f6357-1a58-4f63-9198-c68628df173e from nats_topic, export sends all messages regularly {"level":"debug","message":"Client export-333b6ee8-c3e5-4e61-a090-19cef0030146 connected","ts":"2022-02-09T10:12:58.113449092Z"} {"level":"debug","message":"Published to: channels.f26f6357-1a58-4f63-9198-c68628df173e, payload: [{\"bn\":\"1:\", \"n\":\"exec\", \"vs\":\"ls, -l\"}]","ts":"2022-02-09T10:13:03.420431845Z"}

Do I need to configure something else or add anything more? Is there another way to filter messages that will be sent to cloud?

[mteodor]

is there anything in the logs? log level being set to debug?

[mteodor]

additionally are you using configure.sh?

[marijadstankovic]

I added MF_EXPORT_LOG_LEVEL=debug to export-start script but the log doesn't show anything pi@raspberrypi:~/mainflux/export-master/scripts $ export_start ^[[3~2022/02/09 03:35:06 Configuration loaded from file ./export-config.toml {"level":"info","message":"Export service started, exposed port :8170","ts":"2022-02-09T13:35:06.208525736Z"} {"level":"debug","message":"Client export-333b6ee8-c3e5-4e61-a090-19cef0030146 connected","ts":"2022-02-09T13:35:06.209701699Z"}

[mteodor]

I'm investigating

[marijadstankovic]

I had problems with running configure.sh itself as I'm using windows and some commands weren't recognized, so I followed the script manually. This kind of filtering should work usually? Maybe I didn't build the service correctly. I downloaded export service sometime in November

[mteodor]

it is probably versions mismatch between service itself and cloud services

[marijadstankovic]

hmm ok, thanks

[mteodor]

Hi Marija, first, can you make sure that you are running latest services core services first adjust docker compose to expose nats port 4222 ( explained in docs )

docker-compose --env-file docker/.env -f docker/docker-compose.yml pull
docker-compose --env-file docker/.env -f docker/docker-compose.yml up

bootstrap service

docker-compose --env-file docker/.env -f docker/addons/bootstrap/docker-compose.yml pull
docker-compose --env-file docker/.env -f docker/addons/bootstrap/docker-compose.yml up

provision service edit file docker/addons/provision/configs/config.toml

add

   [bootstrap.content.export]
     file = "marija.toml"

just above [bootstrap.content.export.mqtt]

like this

      cache_url = "localhost:6379"
      cache_pass = ""
      cache_db = "0"

    [bootstrap.content.export]
      file = "marija.toml"

    [bootstrap.content.export.mqtt]
      ca_path = "ca.crt"
      cert_path = "thing.crt"

change type from plain to mfx

docker-compose --env-file docker/.env -f docker/addons/provision/docker-compose.yml pull
docker-compose --env-file docker/.env -f docker/addons/provision/docker-compose.yml up

then create configuration by running

curl -s -S  -X POST  http://localhost:8190/mapping -H "Authorization: $TOK" -H 'Content-Type: application/json'   -d '{"name":"test-provision",  "external_id" : "external_id126", "external_key":"external_key6" }'

$TOK replace with your user token value previously obtained on tokens endpoint now lets move to agent and export make sure that your code is up to date with github upstream on master branch for both repo do

git pull --rebase upstream master

than in export and agent directory build your binaries with just running make

run the agent to retrieve configuration files in agent direcotry run

MF_AGENT_BOOTSTRAP_SKIP_TLS=true \                
MF_AGENT_BOOTSTRAP_ID=external_id126 \
MF_AGENT_BOOTSTRAP_KEY=external_key6 
MF_AGENT_CONFIG_FILE=agent-configs.toml \
MF_AGENT_BOOTSTRAP_URL=http://localhost:8202/things/bootstrap build/mainflux-agent

this will retrieve configuration file both for agent and for export ( for export is saved in marija.toml as previously set)

you need to modify marija.toml

[[routes]]
  mqtt_topic = "channels/<data-channel>/messages"
  nats_topic = "channels.<some_channel>"

to obtain data channel check the bootstrap config

curl -s -S -X GET http://localhost:8202/things/bootstrap/external_id126 -H 'Authorization: external_key6' -H 'Content-Type: application/json'      
{"mainflux_id":"aee25d8e-0e1f-4aed-bc4a-8e8e22cc7369","mainflux_key":"e4799c83-0864-4210-b756-ed6f38dc0d44","mainflux_channels":[{"id":"bdf41428-a262-4bc9-97d9-45942a9198b5","name":"control-channel","metadata":{"type":"control"}},{"id":"e22051cf-0ef2-4ff7-a41d-49e283834b3d","name":"data-channel","metadata":{"type":"data"}}],"content":"{\"agent\":{\"edgex\":{\"url\":\"http://localhost:48090/api/v1/\"},\"heartbeat\":{\"interval\":\"30s\"},\"log\":{\"level\":\"info\"},\"mqtt\":{\"mtls\":false,\"qos\":0,\"retain\":false,\"skip_tls_ver\":true,\"url\":\"localhost:1883\"},\"server\":{\"nats_url\":\"localhost:4222\",\"port\":\"9000\"},\"terminal\":{\"session_timeout\":\"30s\"}},\"export\":{\"exp\":{\"cache_db\":\"0\",\"cache_pass\":\"\",\"cache_url\":\"localhost:6379\",\"log_level\":\"debug\",\"nats\":\"nats://localhost:4222\",\"port\":\"8172\"},\"file\":\"mirko.toml\",\"mqtt\":{\"ca_path\":\"ca.crt\",\"cert_path\":\"thing.crt\",\"channel\":\"\",\"host\":\"tcp://localhost:1883\",\"mtls\":false,\"password\":\"\",\"priv_key_path\":\"thing.key\",\"qos\":0,\"retain\":false,\"skip_tls_ver\":false,\"username\":\"\"},\"routes\":[{\"mqtt_topic\":\"\",\"nats_topic\":\"*\",\"subtopic\":\"\",\"type\":\"mfx\",\"workers\":10}]}}"}

and run export with pointing to the file

MF_EXPORT_CONFIG_FILE=./marija.toml mainflux-export

so when you publish to channel <some_channel> via mqtt data will be published to channels/<data-channel>/messages/channels/<some_channel>

if you havent pull new images for a long time your docker volumes may be initialized with an old version of services and it may be incomaptibile you will have to clean up volumes and start from fresh database

[marijadstankovic]

I tried this but it doesn't work so I'll have to try with cleaning up volumes and fresh db :( only I didn't use just make for agent and export, I used GOOS=linux GOARCH=arm GOARM=7 make since I'm using raspberry4 for the edge