thp / urlwatch

Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
https://thp.io/2008/urlwatch/
Other
2.79k stars 350 forks source link

--test-filter works but not with a normal execution #825

Closed SimoneFelici closed 2 weeks ago

SimoneFelici commented 2 weeks ago

I just set up urlwatch with 4 urls. My first one went great, and also the second one, here is my urls.yaml:

name: "Ransomfeed"
kind: url
url: "https://www.ransomfeed.it/?country=ITA"
filter:
  - xpath: "//div[@class='table-responsive']//tbody/tr[1]"
  - html2text
  - strip

---

name: "Hacker News"
kind: url
url: "https://thehackernews.com/"

filter:
  - xpath: "//div[@class='body-post clear'][1]"
  - html2text
  - shellpipe: "tr -cd '\\11\\12\\15\\40-\\176'"
  - strip

---

name: "Red Hot Cyber"
kind: url
url: "https://www.redhotcyber.com/"
filter:
  - xpath: "(//article[contains(@class, 'elementor-post')])[1]"
  - html2text
  - strip

---

name: "Commissariato di PS Notizie"
kind: url
url: "https://www.commissariatodips.it/notizie/index.html"
filter:
  - xpath: "//div[@class='media article articletype-0 topnews dotted-h'][1]"
  - html2text
  - strip

I just want to use urlwatch for when specific sites publuish a new article.

The thing is, that this 4 urls that I set up they all pass the test:

user@ubuntu:~$ urlwatch --test-filter 1
16078
2024-06-24
                15:33:25
                Compagnia Trasporti Integrati S.R.L
                monti
                Italy
user@ubuntu:~$ urlwatch --test-filter 2
New MOVEit Transfer Vulnerability Under Active Exploitation - Patch ASAP!
Jun 26, 2024
Vulnerability / Data Protection
 A newly disclosed critical security flaw impacting Progress Software MOVEit Transfer is already seeing exploitation attempts in the wild shortly after details of the bug were publicly disclosed. The vulnerability, tracked as CVE-2024-5806 (CVSS score: 9.1), concerns an authentication bypass that impacts the following versions - From 2023.0.0 before 2023.0.11 From 2023.1.0 before 2023.1.6, and From 2024.0.0 before 2024.0.2 "Improper authentication vulnerability in Progress MOVEit Transfer (SFTP module) can lead to Authentication Bypass," the company said in an advisory released Tuesday. Progress has also addressed another critical SFTP-associated authentication bypass vulnerability (CVE-2024-5805, CVSS score: 9.1) affecting MOVEit Gateway version 2024.0.0. Successful exploitation of the flaws could allow attackers to bypass SFTP authentication and gain access to MOVEit Transfer and Gateway systems. watchTowr Labs has since published additional technical specifi
user@ubuntu:~$ urlwatch --test-filter 3
Cybercrime e Dark Web
150.000 dollari. Il costo di uno 0-Day UAF nel Kernel Linux sul Dark Web
Recentemente è emerso un allarme nel mondo della sicurezza informatica: un attore malintenzionato ha annunciato la vendita di una vulnerabilità 0-Day di tipo Use After Free (UAF) nel kernel Linux su un noto forum del dark web. Questa vulnerabilità, se sfruttata, permetterebbe l’esecuzione di codice con privilegi elevati, rappresentando una
RHC Dark Lab
26/06/2024
16:20
user@ubuntu:~$ urlwatch --test-filter 4
25.06.2024
                                                    POLIZIA DI STATO E ANCI PIEMONTE: PATTO PER LA CYBER SICUREZZA
                                                     È stato siglato presso la Questura di Torino il Protocollo d’Intesa tra il Centro Operativo Sicurezza Cibernetica della Polizia Postale Piemonte e...

But only the first and the third one actually send me something to discord(I used a discord webhook for reporting) And also the console shows this:

user@ubuntu:~$ urlwatch
===========================================================================
01. NEW: Commissariato di PS Notizie
===========================================================================

---------------------------------------------------------------------------
NEW: Commissariato di PS Notizie ( https://www.commissariatodips.it/notizie/index.html )
---------------------------------------------------------------------------

-- 

And this is what I got from Discord:

===========================================================================
01. NEW: Commissariato di PS Notizie
===========================================================================

---------------------------------------------------------------------------
NEW: Commissariato di PS Notizie ( https://www.commissariatodips.it/notizie/index.html )
---------------------------------------------------------------------------

-- 
urlwatch 2.28, Copyright 2008-2023 Thomas Perl
Website: https://thp.io/2008/urlwatch/
Support urlwatch development: https://github.com/sponsors/thp
watched 4 URLs in 0 seconds

I don't know if I am missing out on something important but I can't seem to figure out the issue by looking at the wiki

SimoneFelici commented 2 weeks ago

image Photo for reference

Jamstah commented 2 weeks ago

Its only going to send new messages to discord if it detects changes.

If you want to reset the cache so it starts again, delete ~/.cache/urlwatch/cache.db. The next time you run it you should see all four new entries in discord, then it will go back to only sending changes.

SimoneFelici commented 2 weeks ago

Its only going to send new messages to discord if it detects changes.

If you want to reset the cache so it starts again, delete ~/.cache/urlwatch/cache.db. The next time you run it you should see all four new entries in discord, then it will go back to only sending changes.

Now I am on my work machine, and it works, yesterday I was with my pc at home, and I tried to change the Xpath, so it would find a change and send a notification, but all it did was what I showed, maybe I did something wrong in the setup, thank you for your help

SimoneFelici commented 2 weeks ago

To add to this issue, I previously used Discord, I think what was happening is that the text limit of discord cropped the message, Now I am using telegram and works fine.