[feature] Notification for all types of errors (or every time a check has failed)

[kingp0dd]

v0.43

If I remember correctly, I receive xpath errors notifications. However, I noticed that I had other types of error this morning which I only found when looking at the ChangeDetection homepage:

Sample errors:

1. `WebSocket error: getaddrinfo ENOTFOUND playwright-chrome =========================== logs ===========================

ws://playwright-chrome:3000/?stealth=1&--disable-web-security=true error getaddrinfo ENOTFOUND playwright-chrome ws://playwright-chrome:3000/?stealth=1&--disable-web-security=true getaddrinfo ENOTFOUND playwright-chrome ws://playwright-chrome:3000/?stealth=1&--disable-web-security=true code=1006 reason= ============================================================` 2.  ` Error - 404 (Page not found) received ` 3.  (Wait seconds before extracting text is already 1minute) And I didn't receive a notification. Is there an option that I didn't see to enable notification on `all` errors that fail the change detection? I think it's crucial to know if the checks are failing, else I would think that all is fine.

[dgtlmoon]

Agreed, should be two settings

• Number of times a filter can be missing before triggering a notice/message ( 6 by default )
• Number of times any other error can occur before triggering a notice/message ( maybe 10 ? )

[jsnfwlr]

It would be great to have this, but with the option to "notify only once after x-number of errors/missing-filters occur" and "repeat notification every x-number of triggers"

So if you have your watch set to notify after 10 consecutive occurrences of failures", and "repeat notification every x-number of triggers " set to 2, the failure would have to occur 10 times (consecutively) to get the first notification, then you would get another notification after 30 total consecutive occurrences (2 × 10 = 20, plus the first 10), and again at 50, 70, 90 etc.

Some more nice-to-have options that go along with these:

• a setting to notify when a watch stops failing after an fail-streak (extra useful in combination with the previously suggested "notify only once after x-number failures occur" option)
• a setting to notify of x-number of errors over y-duration
• an option to backoff checks by n-factor when x-number of consecutive failure are encountered, then return to the default duration Examples:
  • each time 10 consecutive failures occur, double the amount of time between checks (so a check-time of 30 minutes would go to 1 hour, then 2 hours, 4 hours, etc)
  • each time 10 consecutive failures occur, add 30 minutes to the time between checks (so a check-time of 30 minutes would go to 1 hour, then 90 minutes, then 2 hours, etc)
• an option to delay considering a watch as not-failing until x-number of consecutive checks have succeeded
• an option to pause failure notifications after x-number of notifications and then resume once the failure has been resolved

Admittedly these enhancements are not trivial and would take the functionality of CD closer to a service monitor like uptime-kuma or statping, but they have their uses for monitoring stock levels or item prices - ie when a site can't keep up with the traffic after a product is back in stock or drops in price.

[c0d3z3r0]

+1

Maybe first add a simple version like mentioned above https://github.com/dgtlmoon/changedetection.io/issues/1678#issuecomment-1711353167 and extend it later?

Since I really needed this, I added this locally without any options or magic:

diff --git a/changedetectionio/update_worker.py b/changedetectionio/update_worker.py
index 1cfc0a8..5f6fa30 100644
--- a/changedetectionio/update_worker.py
+++ b/changedetectionio/update_worker.py
@@ -138,6 +138,31 @@ class update_worker(threading.Thread):

         return queued

+    def send_error_notification(self, watch_uuid):
+
+        watch = self.datastore.data['watching'].get(watch_uuid)
+        if not watch:
+            return
+
+        n_object = {'notification_title': 'Changedetection.io - Alert - Error',
+                    'notification_body': "Error for {{{{watch_url}}}}\n\nLink: {{{{base_url}}}}/edit/{{{{watch_uuid}}}}\n".format(
+                        ", ".join(watch['include_filters'])),
+                    'notification_format': 'text'}
+
+        if len(watch['notification_urls']):
+            n_object['notification_urls'] = watch['notification_urls']
+
+        elif len(self.datastore.data['settings']['application']['notification_urls']):
+            n_object['notification_urls'] = self.datastore.data['settings']['application']['notification_urls']
+
+        # Only prepare to notify if the rules above matched
+        if 'notification_urls' in n_object:
+            n_object.update({
+                'watch_url': watch['url'],
+                'uuid': watch_uuid,
+                'screenshot': None
+            })
+            self.notification_q.put(n_object)

     def send_filter_failure_notification(self, watch_uuid):

@@ -286,6 +311,8 @@ class update_worker(threading.Thread):
                         if e.page_text:
                             self.datastore.save_error_text(watch_uuid=uuid, contents=e.page_text)

+                        self.send_error_notification(uuid)
+
                         self.datastore.update_watch(uuid=uuid, update_obj={'last_error': err_text})
                         process_changedetection_results = False

[libklein]

Hey,

I've prepared a draft PR that implements basic functionality. I'd be happy like to hear your suggestions on what kind of behaviors like "notify only once after x-number of errors/missing-filters occur" and "repeat notification every x-number of triggers" should be supported, and where these settings should live (watch/global).

[jsnfwlr]

I've prepared a draft PR that implements basic functionality.

This is why open source is the way of the future, amazing people making the lives of others better, one PR at a time. Thank you.

"notify only once after x-number of errors/missing-filters occur"

I have some Watches that monitor pages served from containers and if the container is in the process of restarting at the time the Watch checks the page, I get a notification that the page is missing. It would be nice to have a setting that says "only notify me that there is an error (or that the content I am filtering for is missing) after 3 (or any number I enter) checks in a row have returned errors or detect the content is missing and then either stop notifying me until the content is found again or repeat the notification every x-number of triggers (see below)" - the counter should reset if the content is found again

"repeat notification every x-number of triggers"

Following on from above, rather than notify that a Watch is erroring once and stopping, it would be good to have a way to repeat the alerts until the issue that is causing the failure is resolved. I've had a few Watches where I had filters for specific text that permanently got removed from the website, and I accidentally cleared the notification about it and forgot about the issue until I realised weeks or months later that I wasn't getting notifications about changes on the page. This could be avoided by having a setting that says "I want to get notified the first time a change (or failure) is detected, plus every fifth (or any number I enter) time a change is detected, and reset the counter after 24 hours (or any duration I enter)". It would also help reduce notification floods for Watches on websites that update frequently and then stop for a while, like if you're monitoring the stock available for a product that is selling out quickly.

where these settings should live (watch/global).

I think they could be set at a Global level but then have a Watch level setting to chose between "use Global notification schedule" or "Define custom notification schedule" where selecting the latter would allow you to override the Global option.

This would allow you to have:

• the current normal behaviour set at the Global level but then allow specific Watches to use the settings above; or
• a Global setting using the options described above, but specific Watches to use the current normal behaviour.

For example you could have:

• a Global setting that notifies you on every change/error (current behaviour), with a Watch set to only notify you after 3 errors;
• a Global setting that only notifies after 10 consecutive failures, but a Watch specific setting that notifies on every failure; or
• a Global setting that waits until a change has been detected for 3 consecutive checks before notifying you but then notifies for every consecutive check that detects a change, plus a Watch that overrides that to wait for changes to be detected for 3 consecutive checks but then notifies for every third consecutive check that detects a change.

Again, thank you for following up on this idea and helping move it along the path to becoming a reality! I hope you can understand the concept I am trying to describe in comments and that they provide enough information for you to implement it.

[dgtlmoon]

the problem is, how do you monitor that the notifications were received properly? the app never tries to resend a notification if say discord is unreachable or whatever..

[jsnfwlr]

the problem is, how do you monitor that the notifications were received properly? the app never tries to resend a notification if say discord is unreachable or whatever..

That seems like a separate problem to limiting the frequency of notifications - in fact I would say the solution needed for the situation described in this github issue would take place in the changedetection code, while retrying-until-success should be part of the apprise code.

[cpy82]

the problem is, how do you monitor that the notifications were received properly? the app never tries to resend a notification if say discord is unreachable or whatever..

I would be happy if there was simply a separate rss feed with error messages

[DarkLordGMS]

@

Hey,

I've prepared a draft PR that implements basic functionality. I'd be happy like to hear your suggestions on what kind of behaviors like "notify only once after x-number of errors/missing-filters occur" and "repeat notification every x-number of triggers" should be supported, and where these settings should live (watch/global).

Hello @libklein Any update on this? A notification for all types of errors would be very useful.

[dgtlmoon]

@DarkLordGMS please dont spam people, people are donating their time to solve the problems

[libklein]

I've updated the PR:

• Rebased onto master - should be mergable.
• Added notify on failure field to edit form, failure notifications can now be enabled on a tag and watch basis. Should this also be a global setting?

[DarkLordGMS]

It would be extremely useful to have it as a global setting so that you can get a notification if there's an error on any URL instead of enabling it on every single URL. It would be a nightmare to enable it on every single URL if you're watching a lot of different websites. @libklein

[libklein]

I've added the setting to the global config.

[DarkLordGMS]

Awesome! Thank you so much! @libklein Hopefully dgtlmoon will push it to dev or master soon so I can try it.