search

chirpstack / chirpstack-concentratord

Concentrator HAL daemon for LoRa gateways.
https://www.chirpstack.io/
MIT License
70 stars 50 forks source link

Process panicks upon start (Multitech Conduit, mPower 6.3.0) #167

Open nicolas-juteau opened 1 week ago

nicolas-juteau commented 1 week ago

What happened?

Attempted to start chirpstack-concentratord 4.4.1 on a Multitech Conduit (LoRa Card MTAC-LORA-H-915) with mPower 6.3.0 version. The process crashes and does not start.

What did you expect?

I did expect the concentratord to start as it should.

Steps to reproduce this issue

Steps:

  1. Install mPower 6.3.0 on a Multitech Conduit gateway (model used is MTCDT-L4N1)
  2. Install chirpstack-concentratord, following the instructions as indicated in documentation (https://www.chirpstack.io/docs/chirpstack-concentratord/installation/multitech.html)
  3. Attempt to start chirpstack-concentratord and notice it does not fork in the background (process exits)
  4. See the logs / command output

Could you share your log output?

root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2# /opt/chirpstack-concentratord-sx1301/chirpstack-concentratord-sx1301 -c concentratord.toml -c channels.toml -c region.toml
thread 'main' panicked at chirpstack-concentratord-sx1301/src/main.rs:111:80:
called `Result::unwrap()` on an `Err` value: lgw_start failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2# RUST_BACKTRACE=1 /opt/chirpstack-concentratord-sx1301/chirpstack-concentratord-sx1301 -c concentratord.toml -c channels.toml -c region.toml
thread 'main' panicked at chirpstack-concentratord-sx1301/src/main.rs:111:80:
called `Result::unwrap()` on an `Err` value: lgw_start failed

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2# RUST_BACKTRACE=full /opt/chirpstack-concentratord-sx1301/chirpstack-concentratord-sx1301 -c concentratord.toml -c channels.toml -c region.toml
thread 'main' panicked at chirpstack-concentratord-sx1301/src/main.rs:111:80:
called `Result::unwrap()` on an `Err` value: lgw_start failed

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
stack backtrace:
   0:   0x29200c - <unknown>
   1:   0x2dced8 - <unknown>
   2:   0x28ea40 - <unknown>
   3:   0x291e40 - <unknown>
   4:   0x29366c - <unknown>
   5:   0x2932bc - <unknown>
   6:   0x293ba0 - <unknown>
   7:   0x2939fc - <unknown>
   8:   0x292554 - <unknown>
   9:   0x2937c4 - <unknown>
  10:    0x1e8b8 - <unknown>
  11:    0x1ed04 - <unknown>
  12:    0x554f0 - <unknown>
  13:    0x73e18 - <unknown>
  14:    0x81770 - <unknown>
  15:   0x2887b8 - <unknown>
  16:    0x561d4 - <unknown>
root@mtcdt:/var/config/chirpstack-concentratord-sx1301/ap2#

2024-06-24T16:49:22.323623+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Starting Concentratord SX1301 (version: 4.4.1, docs: https://www.chirpstack.io/docs/chirpstack-concentratord/)
2024-06-24T16:49:22.329014+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting spi device path, spidev_path: /dev/spidev32765.2
2024-06-24T16:49:22.334618+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting board configuration, lorawan_public: true, clock_source: 0
2024-06-24T16:49:22.341344+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting up concentrator radios
2024-06-24T16:49:22.345681+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring radio, radio: 0, enabled: true, center_freq: 902700000, type: SX1257
2024-06-24T16:49:22.347451+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring radio, radio: 1, enabled: true, center_freq: 903700000, type: SX1257
2024-06-24T16:49:22.349203+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Setting up concentrator channels
2024-06-24T16:49:22.351008+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 0, enabled: true, freq: 902300000, rf_chain: 0, if_freq: -400000
2024-06-24T16:49:22.353912+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 1, enabled: true, freq: 902500000, rf_chain: 0, if_freq: -200000
2024-06-24T16:49:22.355835+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 2, enabled: true, freq: 902700000, rf_chain: 0, if_freq: 0
2024-06-24T16:49:22.357630+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 3, enabled: true, freq: 902900000, rf_chain: 0, if_freq: 200000
2024-06-24T16:49:22.359434+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 4, enabled: true, freq: 903100000, rf_chain: 0, if_freq: 400000
2024-06-24T16:49:22.361229+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 5, enabled: true, freq: 903300000, rf_chain: 1, if_freq: -400000
2024-06-24T16:49:22.363027+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 6, enabled: true, freq: 903500000, rf_chain: 1, if_freq: -200000
2024-06-24T16:49:22.365055+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring multi-SF LoRa channel, channel: 7, enabled: true, freq: 903700000, rf_chain: 1, if_freq: 0
2024-06-24T16:49:22.366720+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring Std LoRa channel, enabled: true, freq: 903000000, rf_chain: 0, if_freq: 300000
2024-06-24T16:49:22.368524+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Configuring FSK channel, enabled: false, freq: 0, rf_chain: 0, if_freq: 0
2024-06-24T16:49:22.370096+00:00 mtcdt chirpstack-concentratord-sx1301[3828]: Starting the concentrator

_spidevpath: /dev/spidev32765.2 looks odd to me.

Your Environment

Component Version
Concentratord 4.4.1
nicolas-juteau commented 1 week ago

FYI: I have tried with other concentratord versions, down to 4.1.0 and the same issue occurs anyway (process panick + same log message with strange spidev path). I am concerned about this because in an another ticket I already managed to run the chirpstack-concentratord 4.3.4 successfully on a Multitech Conduit mPower 6.3.1.

Nevertheless, the issue experienced in this ticket is odd and I think it deserves attention.

nicolas-juteau commented 1 week ago

Also, I am able to run the embedded lora-network-server + packet forwarder provided by the mPower, so it is apparently not a LoRa card issue.

nicolas-juteau commented 1 week ago

After some digging, it seems that since mLinux/mPower 6.3.1 the /dev/spidev mapping changed and makes /dev/spidev32765.2 unusable by chirpstack-concentratord. Indeed:

mLinux/mPower 5.3.8-s1 (concentratord works):

root@mtcdt:/dev# ls -al|grep spidev
lrwxrwxrwx    1 root     root            18 Jun 26 13:53 spidev0.0 -> /dev/spidev32765.2
crw-------    1 root     root      153,   1 Jun 26 13:53 spidev32765.2
crw-------    1 root     root      153,   0 Jun 26 13:53 spidev32766.2

mLinux/mPower 6.3.1 (concentratord will not work):

mtcdt:/dev# ls -al|grep spidev
crw-------    1 root     root      153,   2 Jun 26  2024 spidev0.0
crw-------    1 root     root      153,   1 Jun 26  2024 spidev0.1
crw-------    1 root     root      153,   0 Jun 26  2024 spidev0.2
crw-------    1 root     root      153,   5 Jun 26  2024 spidev1.0
crw-------    1 root     root      153,   4 Jun 26  2024 spidev1.1
crw-------    1 root     root      153,   3 Jun 26  2024 spidev1.2

No wonder that if I manually create a symlink on /dev/spidev32765.2 that maps to /dev/spidev1.2, chirpstack-concentratord will start successfully.

I'm still confused though as to why I have previously been able to run chirpstack-concentratord on a conduit mPower 6.3.1 (https://github.com/chirpstack/chirpstack-mqtt-forwarder/issues/47).

Can somebody confirm the spidev mapping changed on a fresh-from-factory mPower 6.3.1 conduit?

EDIT: corrected mPower version

nicolas-juteau commented 1 week ago

Appears to be confirmed in the mLinux changelog, but since 6.0.1 version: image

Looks like concentratord will need a way to specify the correct spidev path to use depending on the firmware version of the conduit as well as which ap (ap1, ap2) to target