Feature request: Extension of regex filtering to extract data

[f0sh]

I couldn't find a way, how to use re.sub to extract data as a filter in urlwatch. Unless I haven't overseen anything, and there is a way to do it, I would like to open a feature request.

According to #401 I would like to request an extensions of the regex filtering. Instead of replacing matched strings, it would be useful to have a positive regex filter, to extract specific data. An example of this filtering is even in the documentation. However there it is implemented using shellpipes and grep, but for me it turned out to be a bit unstable.

url: https://example.net/shellpipe-grep.txt
filter:
  - shellpipe: "grep -i -o 'price: <span>.*</span>'"

IMO better approach would be to have a filter like re.findall which goes along with re.sub:

url: https://example.net/pricelist.html
filter:
  - re.findall: 'price: <span>(.*)</span>'

[Jamstah]

grep and grepi can be used directly, so you could do something like:

filter:
  - grepi: 'price: <span>.*</span>'
  - re.sub:
      pattern: '^.*(price: <span>.*</span>).*$'
      repl: '\1'

[Jamstah]

findall might be easier though, what are you thinking for the output, just put each match on a new line?

[f0sh]

findall might be easier though, what are you thinking for the output, just put each match on a new line?

yes, that's what I was thinking too.

I didn't check in the source yet, how it is implemented, but it felt like, it could be easier integrated. But maybe the similar names of re.sub in urlwatch and the re package fooled me.

[Jamstah]

Yes, its not hard to add it. A little more than re.sub because you have to do something with the matches, where re.sub will just give you the string to return.

https://github.com/thp/urlwatch/blob/master/lib/urlwatch/filters.py#L831

[Jamstah]

I actually have a couple of places that this would simplify my filters, so have put in an implementation in #805. See what you think.

[thp]

For filtering out HTML elements, use the CSS or XPath filters. Never use regex.

[f0sh]

For filtering out HTML elements, use the CSS or XPath filters. Never use regex.

For me, this was not the intention here. It's more that you want to extract certain data. I just tried to make my problem more clear by taking the previous example from the urlwatch docs. The scenario is more, that you have an p element and want to extract some data from there. E.g [...] and therefore for blablabla we set the price of 2.39€.[...]. The idea is to only grab the data 2.39€ without the whole text.

With re.sub you always have to build a regex which catches the whole paragraph which is error prone. And the grep solution only works on full lines.

I actually got inspired by changedetection.io which I tried recently, because of their GUI and they have this nice data extraction feature. However their scripting is much more troublesome so I would like to stick with urlwatch.

OT: It's just a bit frustrating, why open source often has to invent new wheels instead of joining forces. It would be amazing to see, if changedetection.io would have used urlwatch under the hood, to build a more powerful solution.