Detect and remember specific error types per Host

[Apollon77]

There are several types of errors that show host issues that are only visible in logs right now and many users do not see them. We should try to detect and collected them and allow to display them in Admin.

After some thoughts I would have an proposal for discussion:

• The js-controller already monitors the adapter processes and parses the sterr output
• We define error categories and regex on how to parse exception texts to detect, proposal like:
  • File System/IO errors: EROFS, EIO, EMFILE, EBADF, ENFILE, "Unknown system error -117 ... write"
  • No disk space errors: ENOSPC
  • Free RAM issues: ENOMEM
  • Also: "buffer allocation failed" irrelevant of "code" (is partly RangeError)
  • Access errors: EACCESS, EPERM (??)
  • Remote Hosts errors: EHOSTDOWN (??)
  • File/Dir not exists: ENOENT (??)
  • We define that in a new file (errorDetection.js or such) and for each category we have a list of regex and an english description text (which can be later used in admin if translation is missing)
• We add an object in js-controller to remember the last occurences of such errors by category and adapter instance on that host. Idea would be to remember the last 5-10 errors and could store the full "catched by controller" error data
  • When the js-controller process ends we try to store these information ina file and load on next start. Make sure to add error handling to write because on ENOSPC errors we most likely can not store it at all, but we should try
• as soon as such an error happends and the object is not empty we store the information in a new system.host.x state - wie could just do a "hasErrors" boolean flag or a json state with the list of error types or such. the state needs to be initialized correctly on controller start depending on loaded data
  • we could also store such an info state on system.adapter.x to also know it by instance to maybe show an info in Admin
• We add new messages to controller like "getErrorDetails" (all, or able to filter by category and/or instance?) and "clearErrorDetails" (maybe also all vs b category and/or instance) to access and clear the stored error information

After that we can add a popup mechanism to admin as soon as one host saw such errors and allows to clear them after the user solved them.

[foxriver76]

We define error categories and regex on how to parse exception texts to detect, proposal like:

Not sure, if we really need regex, because I don't know how consistently e.code is used but normally we could check exactly for this, like https://github.com/ioBroker/ioBroker.js-controller/blob/765f7526a5371cc1fe6053189033f390d43c7e50/lib/logger.js#L168-L169

const fs = require('fs');

try {
    fs.mkdirSync('/dev/test/test')
} catch (e) {
    console.log(e.code)
}

results in ENOENT

const fs = require('fs');

try {
    fs.mkdirSync('/dev/test')
} catch (e) {
    console.log(e.code)
}

results in EACCESS, so I guess it should work in all cases.

[Apollon77]

Yes but the real exception is only available in the adapter process ... but when adapter crahses we pot. can not communicate to the controller anymore ... and the js.controller so only "sees the "logged exception". So yes we can make sure that in adapter.js the Exception code is logged in a way we can easiely detect it, but in fact it is string parsing in js.controller

[Apollon77]

I wopuld have an "enhanced" idea for this that I would like to share. I think effort wise it is not much more.

In fact we had the discussion some time ago how to better allow "notifications" in ioBroker and such, also for cases where user want to get informed on "dead devices not sending updates anymore" or such.

So here the "enhanced idea" (even if we might only implement it partly right now):

We create a new "notification" component in iobroker. A notification has:

• a scope: Here we predefine "system" for the cases we planned the feature o our side. But I would also see cases like "we stopped adapter x because crash loop" also as a use case. In Admin we would maybe only add to show "system" scope notifications
• a severity: we can use "error", "warn" and "info"; if we want to have a different wording then logs we can also use "alert", "notice" or such; this will be used to "show" the notifications differently
• a category: For predefined types we can have a category (Memory-Issue, "IO issue" ... kind iof the types above. The categories are scope specific and can contain a localized description
• a message: This is the message of one specific notification; and we store up to X per Scope and category

In fact it is very comparable to above, just with allowing multiple scopes instead of only one.

We enhance the io-package to allow the definition of scopes and their details and also the js-controller defines the ones listed above there. That structure in io-package could be like:

{
    "notifications": [
        {
            "id": "system",
            "name": { // can be ommitted when scope is already existing, then id is shown as name
                "en": "System Notifications",
                "de": "System-Benachrichtigungen"
            },
            "description": { // can be ommitted when scope is already existing, then no description is shown
                "en": "These notifications are collected by the ioBroker system and point to issues you should check and fix",
                "de": "..."
            },
            "categories": [
                {
                    "id": "memIssues",
                    "name": { // can be ommitted when scope is already existing, then id is shown as name
                        "en": "Issues with RAM availability",
                        "de": "Probleme mit der Speicher-Verfügbarkeit"
                    },
                    "description": { // can be ommitted when scope is already existing, then no description is shown
                        "en": "...",
                        "de": "..."
                    },
                    "regex": [ // optional if auto detection method should be used
                        "^Exception-Code: ENOMEM",
                        "..."
                    ],
                    "count": 10 // how many mesaages of this category should be remembered, default 10 or such
                }
            ]
        }
    ]
}

For the notification system we create a simply class that holds the logic, gets initialized by controller (needs to read system.adapter.name objects for all installed adapters). Then the controller gets new messages (as defineed above) to read these data and also a message to send in a new notification. The controller code itself can directly use the class to send in the error message strings and so the regex stuff is checked and a new mesage is registered automatically.

Each system.adapter.host gets a json state per scope with a "number of each category" json struct as summary like system.host.myhost.notifications.system = {"memIssues": 5, "someOtherIssue": 10} (or we also add besid ethe number the "last timestamp" per category like {"memIssues": {"count": 5, last: 12234566565}, "someOtherIssue": {...}} )

With this we can exactly do what we want, BUT also (in future) an adapter could define an own scope and even an other adapter (a notification) adapter can also listen to all these states and send out SMS or such ...

[GermanBluefox]

Sounds good. But I would store additionally to RAm the messages as a JSON state to avoid lost of messages by crash

[Apollon77]

The idea was to persis the "host collected stuff" in a file and store on end/crash and load on controller start ... We could also use a json sta ebut could be much data