apache / incubator-stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm
https://stormcrawler.apache.org/
Apache License 2.0
887 stars 262 forks source link

Update jsoup from 1.14.3 to 1.15.1 #968

Closed rzo1 closed 2 years ago

rzo1 commented 2 years ago

Bumps jsoup from 1.14.3 to 1.15.1.

Release notes

Sourced from jsoup's releases.

jsoup 1.15.1 is out now with a bunch of improvements and bug fixes.

Changelog

Sourced from jsoup's changelog.

jsoup changelog

*** Release 1.15.1 [2022-May-15]

  • Change: removed previously deprecated methods and classes (including org.jsoup.safety.Whitelist; use org.jsoup.safety.Safelist instead).

  • Improvement: when converting jsoup Documents to W3C Documents in W3CDom, preserve HTML valid attribute names if the input document is using the HTML syntax. (Previously, would always coerce using the more restrictive XML syntax.) jhy/jsoup#1648

  • Improvement: added the :containsWholeText(text) selector, to match against non-normalized Element text. That can be useful when elements can only be distinguished by e.g. specific case, or leading whitespace, etc. jhy/jsoup#1636

  • Improvement: added Element#wholeOwnText() to retrieve the original (non-normalized) ownText of an Element. Also added the :containsWholeOwnText(text) selector, to match against that. BR elements are now treated as newlines in the wholeText methods. jhy/jsoup#1636

  • Improvement: added the :matchesWholeText(regex) and :matchesWholeOwnText(regex) selectors, to match against whole (non-normalized, case sensitive) element text and own text, respectively. jhy/jsoup#1636

  • Improvement: when evaluating an XPath query against a context element, the complete document is now visible to the query, vs only the context element's sub-tree. This enables support for queries outside (parent or sibling) the element, e.g. ancestor-or-self::*. jhy/jsoup#1652

  • Improvement: allow a maxPaddingWidth on the indent level in OutputSettings when pretty printing. This defaults to 30 to limit the indent level for very deeply nested elements, and may be disabled by setting to -1. jhy/jsoup#1655

  • Improvement: when cloning a Node or an Element, the clone gets a cloned OwnerDocument containing only that clone, so as to preserve applicable settings, such as the Pretty Print settings. jhy/jsoup#763

  • Improvement: added a convenience method Jsoup.parse(File). jhy/jsoup#1693

  • Improvement: in the NodeTraversor, added default implementations for NodeVisitor.tail() and NodeFilter.tail(), so that code using only head() methods can be written as lambdas.

  • Improvement: in NodeTraversor, added support for removing nodes via Node.remove() during NodeVisitor.head(). jhy/jsoup#1699

  • Improvement: added Node.forEachNode(Consumer) and Element.forEach(Consumer<Element) methods, to efficiently traverse the DOM with a functional interface. jhy/jsoup#1700

  • Bugfix: boolean attribute names should be case-insensitive, but were not when the parser was configured to preserve

... (truncated)

Commits


jnioche commented 2 years ago

thanks @rzo1 did you check that the Xsoup based resources in https://github.com/DigitalPebble/storm-crawler/tree/master/core/src/main/java/com/digitalpebble/stormcrawler/jsoup still work after the update?

rzo1 commented 2 years ago

@jnioche I did run JSoupFiltersTest after the upgrade, which didn't break. In addition, I conducted an upgrade of JSoup in Xsoup locally and did run their unit tests (related PR is now merged), which didn't break either. It seems, they did not depend on any removed classes / methods (no compile or runtime issues on the Xsoup site).

Therefore, I am confident, that it does not break anything but might enable new XPath-based use-cases :)

rzo1 commented 2 years ago

We might want to add an exclusion block for jsoup in the xsoup declaration to avoid pulling 1.13.x (which is contained in the latest released Xsoup version) and omit the conflict warning with 1.14.3 or (now) 1.15.1).

Wdyt?

jnioche commented 2 years ago

We might want to add an exclusion block for jsoup in the xsoup declaration to avoid pulling 1.13.x (which is contained in the latest released Xsoup version) and omit the conflict warning with 1.14.3 or (now) 1.15.1).

Wdyt?

Excellent idea!

rzo1 commented 2 years ago

Ok - will add the exclusion later.