search

brocaar / chirpstack-network-server

ChirpStack Network Server is an open-source LoRaWAN network-server.
https://www.chirpstack.io
MIT License
1.49k stars 546 forks source link

redis questions #512

Open chopmann opened 3 years ago

chopmann commented 3 years ago

What happened?

redis keys seem to disappear or are not updated.

What did you expect?

redis key to live as long as the ttl and get updated

Steps to reproduce this issue

Steps:

  1. deploy redis (sentinel, cluster or master-replica)
  2. let the network-server run
  3. for keys not being updated: replace the device-profile of device(s)

Could you share your log output?


chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:37Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:000732fffe6b9be6"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:39Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0003abe"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:39Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:7076ff00560504fa"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:41Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0003d43"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:41Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0004632"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:43Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a00026a7"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:46Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0003d42"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:57Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0004633"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:58Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:7076ff00640306bc"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:28:59Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a00037b2"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:05Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a000419a"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:07Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a000419a"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:07Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:000732fffe6b9be6"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:08Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0003abe"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:08Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0003d42"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:09Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:7076ff00560504fa"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:10Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:aa555a0000000101"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:11Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:aa555a0000000101"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:14Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0003d43"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:14Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0004632"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:14Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a00026a7"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:24Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a0004633"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:28Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:7076ff00640306bc"
chirpstack-ns-5d6b49bdc6-rq2nk chirpstack-ns time="2020-11-27T14:29:30Z" level=error msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a00037b2"

I added the log output to the network-server where redis.nil was being checked. The update bug we found out yesterday while updating some devices and they started failing after that. We checked the database and finally redis, we noticed that the device-profile was wrong, after modifying the value with the right device-profile-id the devices started working.

We have 3 NS and 3 AS. I reduced the count to 1 and the Issue still appears. Is this expected?

Your Environment

Component Version
Application Server 3.12.2
Network Server 3.11.0
Gateway Bridge
Chirpstack API
Geolocation
Concentratord
brocaar commented 3 years ago

I'm not exactly sure if I understand the issue.

replace the device-profile of device(s)

What did you expect after this and what did not work?

msg="redis key does not exist" error="redis: nil" redis_key="lora:ns:gw:00800000a00037b2"

I'm not sure to which part of the code you added this log and what the relation is with the device that you are updating.