search

chirpstack / chirpstack-concentratord

Concentrator HAL daemon for LoRa gateways.
https://www.chirpstack.io/
MIT License
76 stars 56 forks source link

Process panicks upon start (Multitech Conduit, mPower 6.3.0) #167

Closed nicolas-juteau closed 2 months ago

nicolas-juteau commented 5 months ago

What happened?

Attempted to start chirpstack-concentratord 4.4.1 on a Multitech Conduit (LoRa Card MTAC-LORA-H-915) with mPower 6.3.0 version. The process crashes and does not start.

What did you expect?

I did expect the concentratord to start as it should.

Steps to reproduce this issue

Steps:

  1. Install mPower 6.3.0 on a Multitech Conduit gateway (model used is MTCDT-L4N1)
  2. Install chirpstack-concentratord, following the instructions as indicated in documentation (https://www.chirpstack.io/docs/chirpstack-concentratord/installation/multitech.html)
  3. Attempt to start chirpstack-concentratord and notice it does not fork in the background (process exits)
  4. See the logs / command output

Could you share your log output?

root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2# /opt/chirpstack-concentratord-sx1301/chirpstack-concentratord-sx1301 -c concentratord.toml -c channels.toml -c region.toml
thread 'main' panicked at chirpstack-concentratord-sx1301/src/main.rs:111:80:
called `Result::unwrap()` on an `Err` value: lgw_start failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2# RUST_BACKTRACE=1 /opt/chirpstack-concentratord-sx1301/chirpstack-concentratord-sx1301 -c concentratord.toml -c channels.toml -c region.toml
thread 'main' panicked at chirpstack-concentratord-sx1301/src/main.rs:111:80:
called `Result::unwrap()` on an `Err` value: lgw_start failed

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2# RUST_BACKTRACE=full /opt/chirpstack-concentratord-sx1301/chirpstack-concentratord-sx1301 -c concentratord.toml -c channels.toml -c region.toml
thread 'main' panicked at chirpstack-concentratord-sx1301/src/main.rs:111:80:
called `Result::unwrap()` on an `Err` value: lgw_start failed

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
stack backtrace:
   0:   0x29200c - <unknown>
   1:   0x2dced8 - <unknown>
   2:   0x28ea40 - <unknown>
   3:   0x291e40 - <unknown>
   4:   0x29366c - <unknown>
   5:   0x2932bc - <unknown>
   6:   0x293ba0 - <unknown>
   7:   0x2939fc - <unknown>
   8:   0x292554 - <unknown>
   9:   0x2937c4 - <unknown>
  10:    0x1e8b8 - <unknown>
  11:    0x1ed04 - <unknown>
  12:    0x554f0 - <unknown>
  13:    0x73e18 - <unknown>
  14:    0x81770 - <unknown>
  15:   0x2887b8 - <unknown>
  16:    0x561d4 - <unknown>
root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2#

2024-06-24T16:49:22.323623+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Starting Concentratord SX1301 (version: 4.4.1, docs: https://www.chirpstack.io/docs/chirpstack-concentratord/)
2024-06-24T16:49:22.329014+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting spi device path, spidev_path: /dev/spidev32765.2
2024-06-24T16:49:22.334618+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting board configuration, lorawan_public: true, clock_source: 0
2024-06-24T16:49:22.341344+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting up concentrator radios
2024-06-24T16:49:22.345681+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring radio, radio: 0, enabled: true, center_freq: 902700000, type: SX1257
2024-06-24T16:49:22.347451+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring radio, radio: 1, enabled: true, center_freq: 903700000, type: SX1257
2024-06-24T16:49:22.349203+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting up concentrator channels
2024-06-24T16:49:22.351008+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 0, enabled: true, freq: 902300000, rf_chain: 0, if_freq: -400000
2024-06-24T16:49:22.353912+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 1, enabled: true, freq: 902500000, rf_chain: 0, if_freq: -200000
2024-06-24T16:49:22.355835+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 2, enabled: true, freq: 902700000, rf_chain: 0, if_freq: 0
2024-06-24T16:49:22.357630+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 3, enabled: true, freq: 902900000, rf_chain: 0, if_freq: 200000
2024-06-24T16:49:22.359434+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 4, enabled: true, freq: 903100000, rf_chain: 0, if_freq: 400000
2024-06-24T16:49:22.361229+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 5, enabled: true, freq: 903300000, rf_chain: 1, if_freq: -400000
2024-06-24T16:49:22.363027+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 6, enabled: true, freq: 903500000, rf_chain: 1, if_freq: -200000
2024-06-24T16:49:22.365055+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 7, enabled: true, freq: 903700000, rf_chain: 1, if_freq: 0
2024-06-24T16:49:22.366720+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring Std LoRa channel, enabled: true, freq: 903000000, rf_chain: 0, if_freq: 300000
2024-06-24T16:49:22.368524+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring FSK channel, enabled: false, freq: 0, rf_chain: 0, if_freq: 0
2024-06-24T16:49:22.370096+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Starting the concentrator

_spidevpath: /dev/spidev32765.2 looks odd to me.

Your Environment

Component Version
Concentratord 4.4.1
nicolas-juteau commented 5 months ago

FYI: I have tried with other concentratord versions, down to 4.1.0 and the same issue occurs anyway (process panick + same log message with strange spidev path). I am concerned about this because in an another ticket I already managed to run the chirpstack-concentratord 4.3.4 successfully on a Multitech Conduit mPower 6.3.1.

Nevertheless, the issue experienced in this ticket is odd and I think it deserves attention.

nicolas-juteau commented 5 months ago

Also, I am able to run the embedded lora-network-server + packet forwarder provided by the mPower, so it is apparently not a LoRa card issue.

nicolas-juteau commented 5 months ago

After some digging, it seems that since mLinux/mPower 6.3.1 the /dev/spidev mapping changed and makes /dev/spidev32765.2 unusable by chirpstack-concentratord. Indeed:

mLinux/mPower 5.3.8-s1 (concentratord works):

root@mtcdt:/dev# ls -al|grep spidev
lrwxrwxrwx    1 root     root            18 Jun 26 13:53 spidev0.0 -> /dev/spidev32765.2
crw-------    1 root     root      153,   1 Jun 26 13:53 spidev32765.2
crw-------    1 root     root      153,   0 Jun 26 13:53 spidev32766.2

mLinux/mPower 6.3.1 (concentratord will not work):

mtcdt:/dev# ls -al|grep spidev
crw-------    1 root     root      153,   2 Jun 26  2024 spidev0.0
crw-------    1 root     root      153,   1 Jun 26  2024 spidev0.1
crw-------    1 root     root      153,   0 Jun 26  2024 spidev0.2
crw-------    1 root     root      153,   5 Jun 26  2024 spidev1.0
crw-------    1 root     root      153,   4 Jun 26  2024 spidev1.1
crw-------    1 root     root      153,   3 Jun 26  2024 spidev1.2

No wonder that if I manually create a symlink on /dev/spidev32765.2 that maps to /dev/spidev1.2, chirpstack-concentratord will start successfully.

I'm still confused though as to why I have previously been able to run chirpstack-concentratord on a conduit mPower 6.3.1 (https://github.com/chirpstack/chirpstack-mqtt-forwarder/issues/47).

Can somebody confirm the spidev mapping changed on a fresh-from-factory mPower 6.3.1 conduit?

EDIT: corrected mPower version

nicolas-juteau commented 5 months ago

Appears to be confirmed in the mLinux changelog, but since 6.0.1 version: image

Looks like concentratord will need a way to specify the correct spidev path to use depending on the firmware version of the conduit as well as which ap (ap1, ap2) to target

brocaar commented 2 months ago

Sorry for the delay, I haven't had the time yet to validate this. What I will do is update my Conduit to the latest firmware version and then update the SPI dev. paths where needed. Note that in case needed, there is always an option to override this path in the concentratord.toml file (e.g. https://github.com/chirpstack/chirpstack-concentratord/blob/master/chirpstack-concentratord-sx1302/src/config/mod.rs#L109).