Closed kongomongo closed 1 year ago
If it's something that can be done line-based, grepi
is the filter you might want, using regular expressions.
Unfortunately it is impossible using grep :(
Then I found out about
css: selector: "*" exclude: ".ads"
. But this seems to somehow clone the output multiple times and makes the output unusable.
This exclude
subfilter should actually be working. Do you have a minimal example (input / filter / expected output / actual output) that we can use to investigate the issue?
There's even an example in the docs:
Another useful option with XPath and CSS filters is ``exclude``.
Elements selected by this ``exclude`` expression are removed from the
final result. For example, the following job will not have any ``<a>``
tag in its results:
.. code:: yaml
url: https://example.org/css-exclude.html
filter:
- css:
selector: body
exclude: a
And for that one, we even have testcase data:
https://example.org/css-exclude.html:
input: |-
<html>
<body>
<h1>A page in a book</h1>
<p>And some paragraph, too. <a href="http://example.net/">Also check out example.net!</a></p>
</body>
</html>
output: |
<body>
<h1>A page in a book</h1>
<p>And some paragraph, too. </p>
</body>
I guess it might be that selector: "*"
is the problem, and using body
or html
as the selector might be better?
Thanks for the input !
Yes, replacing * with body did the trick! Now nothing gets cloned or otherwise mangled. Awesome.
Just for reference. These were my inputs:
stuff.html:
<div class="thumbnail" tr="Produktliste">
<a href="https://some-site.info/BRAND-Special-modern-type-Product-long-Muster-weiss" name="1263/24/91/40"><img src="/$WS/sitesite/websale8_shop-sitesite/produkte/medien/bilder/klein/OL6WAHM1_M0_1.jpg" title="BRAND Special modern fit Product long Muster wei" alt="BRAND Special modern fit Product long Muster wei" class="artikel"></a>
<div class="nachhaltigkeit-stoerer"></div>
</div>
<div class="product_list_div_content" >
<a href="https://some-site.info/BRAND-Special-modern-type-Product-long-Muster-weiss" class="produktLink pr-name"><span>BRAND Special modern fit Product long Muster wei</span></a>
<a href="https://some-site.info/BRAND-Special-modern-type-Product-long-Muster-weiss" class="produktLink discount_ja sale-stoerer BRAND-stoerer">%</a>
<div class="preisinfo">
<div class='produktbewertung bewertung_OL6WAHM1M0'></div>
<p class="uvpprice">UVP <span class="uvp">69.95 €</span></p>
<p class="price price_sale">54,99 €</p>
</div>
<div class="GroesseVisibleOnHover">
<div class="weitere_farben">
<p>Weitere Farben:</p>
<div class="weitere_farbe">
<a href="https://some-site.info/BRAND-Special-modern-type-Product-long-Muster-navy" itemprop="isSimilarTo"><img src="/$WS/sitesite/websale8_shop-sitesite/produkte/medien/bilder/klein/OL6WAHM1_M6_1.jpg" alt="BRAND Special modern fit Product long Muster navy preisreduziert" title="BRAND Special modern fit Product long Muster navy preisreduziert"></a>
</div>
</div>
</div>
</div>
<br class="clear">
</div>
command: "cat ~/stuff.html"
filter:
- css:
selector: "*"
exclude: ".weitere_farben,div.thumbnail > a"
- html2text:
method: lynx
nolist:
- strip
WRONG output using *
BRAND Special modern fit Product long Muster wei %
UVP 69.95 €
54,99 €
BRAND Special modern fit Product long Muster wei %
UVP 69.95 €
54,99 €
BRAND Special modern fit Product long Muster wei %
UVP 69.95 €
54,99 €
BRAND Special modern fit Product long Muster wei %
UVP 69.95 €
54,99 €
BRAND Special modern fit Product long Muster wei BRAND
Special modern fit Product long Muster wei %
UVP 69.95 €
54,99 €
UVP 69.95 €
69.95 €
54,99 €
correct output using body even though theres no body tag in the input ;-)
BRAND Special modern fit Product long Muster wei %
UVP 69.95 €
54,99 €
Closing this as "works for me" based on your comment.
Hi there,
many thanks for this wonderful tool. I have now come up various times to the following problem:
I'm scraping data as HTML and there are certain things that I want to exclude in the comparison. Mostly it works via by-tag or css or even xpath to select only the relevant portions.
But sometimes it is much harder to select the relevant content (because of fuzzy, generated or changing classes/ids) that to exclude the irrelevant content.
I have two examples. One is a "page footer" that is a
The other example is a dynamically generated list with data that contains advertisements randomly placed within the result. The ads have an easy class to filter them.
I have tried various ways from
xpath '/*[not(self::picture)]'
tocss: '*:not(.ads)'
but they don't appear to work correctly because of children in the tree causing the filters to fail.Then I found out about
css: selector: "*" exclude: ".ads"
. But this seems to somehow clone the output multiple times and makes the output unusable.Am I doing it wrong or is there any solution to this?
Should I post examples?