search

CrazyIvan359 / mqttany

MQTTany is designed to make it easy to connect hardware on single board computers to your home automation solution.
https://crazyivan359.github.io/mqttany/index.html
MIT License
6 stars 6 forks source link

I2C unresponsive #118

Closed pawillaert closed 3 years ago

pawillaert commented 3 years ago

Hello CrazyIvan359,

Let me start by thanking you for the development of this nice piece of software and the effort you are putting in! Below is the description of a problem I am running in to. Could you please have a look and let me know how I can further investigate this?

Describe the bug

Unsure if this is a true mqttany bug or if I may have missed something.

Environment: Hardware: Raspberry Pi 3B with 5 MCP23017 modules, connected to relay modules to manage a total of 80 relays which will form the heart of the data-rack I am building.

Software: On the Pi I have installed:

All has been setup and when all is working properly, I can:

The problem I encounter: What I have experienced 3 times now, is that all-of-a-sudden all relays become unresponsive and none of the ways to toggle the relays I mentioned above work anymore. Clearly this is no behavior I'd like to see for a setup that will become the center of my data-rack :-). This feature typically happens overnight, when a long idle period has happened. This morning however all still worked and when I tested again a couple of hours later, nothing worked again.

On a previous occurence restarting mqttany 'solved' the problem. This morning I decided to use the lowest level: i2ctools:

Expected behavior

A stable environment.

Environment

Configuration

I only pasted the i2c configuration of the first mcp23017 module, the other 4 are identical except for their address.

######## I2C Module ########
i2c:

  ### Polling Interval
  polling interval: 60 # interval in seconds to poll all pins

  ### Device Configuration
  # !! this id is used as the node so it must not match any other node or module name !!

  # +--------------------------+
  # | Device with adress: 0x20 |
  # +--------------------------+
  mcp23017-20:

    ### Device Friendly Name
    # Friendly name used in logging etc
    # Substitutions:
    # - {device_id} = name of device's section in config file
    # - {address:02x} = device address in hex (ex. '20')
    # - {device} = device identifier
    name: '{device_id}'

    ### Device Type
    # Device identifier, see the wiki for a list of supported devices
    device: 'mcp23017'

    ### I2C Bus ID
    # ID of the I2C bus to use on your SBC
    # ex. Raspberry Pi 3 this will probably be 1
    # Can also specify full path ex. '/dev/i2c-1'
    bus: 1

    ### Device Address
    # 7-bit hex address of the device (ex. 0x20)
    # this can also be a base-10 integer
    address: 0x20

    ### Device Specific Options
    # Each device type will have some options specific to it
    # consult the wiki page for the device for details.
    mcp230xx:

      pin-a0:
        pin: 00
        name: 'GPIOA0'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a1:
        pin: 01
        name: 'GPIOA1'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a2:
        pin: 02
        name: 'GPIOA2'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a3:
        pin: 03
        name: 'GPIOA3'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a4:
        pin: 04
        name: 'GPIOA4'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a5:
        pin: 05
        name: 'GPIOA5'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a6:
        pin: 06
        name: 'GPIOA6'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-a7:
        pin: 07
        name: 'GPIOA7'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b0:
        pin: 08
        name: 'GPIOB0'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b1:
        pin: 09
        name: 'GPIOB1'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b2:
        pin: 10
        name: 'GPIOB2'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b3:
        pin: 11
        name: 'GPIOB3'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b4:
        pin: 12
        name: 'GPIOB4'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b5:
        pin: 13
        name: 'GPIOB5'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b6:
        pin: 14
        name: 'GPIOB6'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

      pin-b7:
        pin: 15
        name: 'GPIOB7'
        direction: 'output'
        #resistor: 'off'
        invert: true
        initial state: OFF

Logs

I currently am running mqttany as a service with no logging enabled.

2021-05-07 19:18:06,747 [INFO ] [core                    ] MQTTany 0.14.3 starting
2021-05-07 19:18:06,879 [INFO ] [core.gpio               ] Detected board: RASPBERRY_PI_3B
2021-05-07 19:18:07,688 [INFO ] [core                    ] Module 'mqtt' started successfully
2021-05-07 19:18:07,696 [INFO ] [core                    ] Module 'i2c' started successfully
2021-05-07 19:18:07,716 [INFO ] [mqtt                    ] Connected to broker '127.0.0.1:1883'
CrazyIvan359 commented 3 years ago

Thank you for your kind words and detailed description of the issue. I am not aware of any issues like this, but that's not to say it isn't an MQTTany issue.

I'll need you to run MQTTany with verbose logging enabled to find out if this may be an issue with MQTTany or something else.

First, stop the service and manually launch MQTTany with the -vv option. Once it has loaded fully, stop it with Ctrl+C and then paste or attach the log here. Now add the verbose flag to the service file (edit the file and reload the systemctl daemon as per the install instructions), delete your current log file, and start the service again. Once the issue happens again, stop the service and attach that log file here (don't paste it, it will be very very long) and tell me what time frame it happened in (your local time).

Side note: I notice you have taken the long way to setup your pins. I know the concept isn't obvious on first read, but using arrays in your pin configurations will drop you from 16 pin configs to 2.

pin-a:
  pin: [0,1,2,3,4,5,6,7]
  name: 'GPIOA{index}'

pin-b:
  pin: [8,9,10,11,12,13,14,15]
  name: 'GPIOB{index}'

Use the above and add your other pin options to get the same results with a much smaller config file.

pawillaert commented 3 years ago

Thank you for your quick response!

Please find attached the logfile of the -vv start and stop of the application. mqttany start.log

When I modify the service I assume this only needs the -v or does that need the -vv option?

Yes I did read and understood the array concept but started cutting and pasting :-) and in hindsite will reset this to the much shorter and cleaner array format.

CrazyIvan359 commented 3 years ago

Use -vv in the service file as well. A single 'v' is debug logging, 2 'v's is verbose logging.

I'm glad you understood it! I've had several people say they didn't get it and needed examples. It's on my to-do list but I haven't gotten to it.

pawillaert commented 3 years ago

Got it, thank you. Will circle back as and when this happens again.

CrazyIvan359 commented 3 years ago

Sounds good. As expected, I don't see anything out of the ordinary in the startup log.

pawillaert commented 3 years ago

Well... started the application with the -vv key before sending my last post, now 7 days ago. Ever since, the problem has not reoccurred yet. Typically the problem surfaced after a couple of days. Can only hope that enabling the debug doesn't influence the behavior.

Will keep it running for a while; any other suggestions at this point?

CrazyIvan359 commented 3 years ago

Always the way, isn't it... I suggest leaving it on verbose logging. If you're running on the SD card you may want to setup the log in ZRAM to save wear on the card.

As for the -v and -vv flags, they just change the log level and format string. I wrote my own logging class wrappers to support the additional log level and getting the processor identifier, so the actual program doesn't need to concern itself with the log level.

pawillaert commented 3 years ago

Ok, thx for the feedback and suggestions.

CrazyIvan359 commented 3 years ago

@pawillaert did this issue ever happen again?

pawillaert commented 3 years ago

Hi Michael, no since using the debug mode it has not.

Michael Murton schreef op 2021-08-02 16:07:

@pawillaert [1] did this issue ever happen again?

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or unsubscribe [3].

Links:

[1] https://github.com/pawillaert [2] https://github.com/CrazyIvan359/mqttany/issues/118#issuecomment-891057156 [3] https://github.com/notifications/unsubscribe-auth/AUAO5E2ZI7ONINEYQKD5LBDT22RBDANCNFSM44QT7REA

CrazyIvan359 commented 3 years ago

Closing this for now since we have no new data or instances of it happening. If it occurs again or you have new information we can reopen this and keep working on it.