Starting a Counter loading from persistence

[shadetree01010100]

[2017-10-17 15:20:07.113] NIO [ERROR] [test_slack.callCounter] Failed to start
Traceback (most recent call last):
  File "c:\users\tom\nio\nio\nio\util\runner.py", line 94, in do_start
    self.start()
  File "C:\Users\Tom\nio\projects\LOCAL\blocks\counter\counter_block.py", line 67, in start
    self._schedule_reset()
  File "C:\Users\Tom\nio\projects\LOCAL\blocks\counter\counter_block.py", line 121, in _schedule_reset
    self.reset()
  File "C:\Users\Tom\nio\projects\LOCAL\blocks\counter\counter_block.py", line 197, in reset
    self.notify_signals(self.for_each_group(self.reset_group))
  File "c:\users\tom\nio\nio\nio\block\mixins\group_by\group_by.py", line 72, in for_each_group
    result = target(group, *args, **kwargs)
  File "C:\Users\Tom\nio\projects\LOCAL\blocks\counter\counter_block.py", line 203, in reset_group
    key, self._cumulative_count[key])
KeyError

This counter is configured to reset at midnight UTC, the service has missed 4 reset events since it was last run. The service would not start (configure failure because the above block failed to start) until I set "load_from_persistence": false

blocks/etc/persist/test_slack_callCounter.dat:

{
    "_last_reset": {
        "__datetime__": {
            "minute": 0,
            "second": 0,
            "microsecond": 455,
            "month": 10,
            "day": 13,
            "hour": 0,
            "year": 2017
        }
    },
    "_cumulative_count": {
        "null": 15
    },
    "_groups": {
        "__set__": [
            null
        ]
    }
}

[shadetree01010100]

Reproduction:

• Block has null group, as in above persist file
• reset event has been missed
• start service, with block loading from persistence

Removing the null group from the persist file allows service to start normally. I think the addition of this null group is the root cause.

[shadetree01010100]

found the problem, if the group_by string is empty, the persistence file is saving a group with the name null, rather than a string representation of null, which causes an expected key error. entering any arbitrary string for group_by causes the group name to evaluate as a string.

To test this I set my block's group_by to "null", processed a couple signals, and then deleted the group_by value, restarrted and processed more signals. Behold the broken persist file!

    "_cumulative_count": {
        "__dct__": {
            "null": 2,
            "null": 1
        }
    },
    "_last_reset": {
        "__datetime__": {
            "minute": 35,
            "day": 17,
            "microsecond": 10600,
            "month": 10,
            "year": 2017,
            "hour": 16,
            "second": 0
        }
    },
    "_groups": {
        "__set__": [
            null,
            "null"
        ]
    }
}

[shadetree01010100]

this same problem was posted on our forum: https://forum.n.io/discussion/help/i-have-an-interesting-error-being-thrown-for-the-debounce-block-2

[shadetree01010100]

this turns out to be an issue in persistence

[shadetree01010100]

reopening this issue as we add methods to persistence to handle these types of things