Changelog
### 4.11.1
```
This release was done to ensure that the unit tests are packaged along
with the released source. There are no functionality changes in this
release, but there are a few other packaging changes:
* The Japanese and Korean translations of the documentation are included.
* The changelog is now packaged as CHANGELOG, and the license file is
packaged as LICENSE. NEWS.txt and COPYING.txt are still present,
but may be removed in the future.
* TODO.txt is no longer packaged, since a TODO is not relevant for released
code.
```
### 4.11.0
```
* Ported unit tests to use pytest.
* Added special string classes, RubyParenthesisString and RubyTextString,
to make it possible to treat ruby text specially in get_text() calls.
[bug=1941980]
* It's now possible to customize the way output is indented by
providing a value for the 'indent' argument to the Formatter
constructor. The 'indent' argument works very similarly to the
argument of the same name in the Python standard library's
json.dump() function. [bug=1955497]
* If the charset-normalizer Python module
(https://pypi.org/project/charset-normalizer/) is installed, Beautiful
Soup will use it to detect the character sets of incoming documents.
This is also the module used by newer versions of the Requests library.
For the sake of backwards compatibility, chardet and cchardet both take
precedence if installed. [bug=1955346]
* Added a workaround for an lxml bug
(https://bugs.launchpad.net/lxml/+bug/1948551) that causes
problems when parsing a Unicode string beginning with BYTE ORDER MARK.
[bug=1947768]
* Issue a warning when an HTML parser is used to parse a document that
looks like XML but not XHTML. [bug=1939121]
* Do a better job of keeping track of namespaces as an XML document is
parsed, so that CSS selectors that use namespaces will do the right
thing more often. [bug=1946243]
* Some time ago, the misleadingly named "text" argument to find-type
methods was renamed to the more accurate "string." But this supposed
"renaming" didn't make it into important places like the method
signatures or the docstrings. That's corrected in this
version. "text" still works, but will give a DeprecationWarning.
[bug=1947038]
* Fixed a crash when pickling a BeautifulSoup object that has no
tree builder. [bug=1934003]
* Fixed a crash when overriding multi_valued_attributes and using the
html5lib parser. [bug=1948488]
* Standardized the wording of the MarkupResemblesLocatorWarning
warnings to omit untrusted input and make the warnings less
judgmental about what you ought to be doing. [bug=1955450]
* Removed support for the iconv_codec library, which doesn't seem
to exist anymore and was never put up on PyPI. (The closest
replacement on PyPI, iconv_codecs, is GPL-licensed, so we can't use
it--it's also quite old.)
```
### 4.10.0
```
* This is the first release of Beautiful Soup to only support Python
3. I dropped Python 2 support to maintain support for newer versions
(58 and up) of setuptools. See:
https://github.com/pypa/setuptools/issues/2769 [bug=1942919]
* The behavior of methods like .get_text() and .strings now differs
depending on the type of tag. The change is visible with HTML tags
like <script>, <style>, and <template>. Starting in 4.9.0, methods
like get_text() returned no results on such tags, because the
contents of those tags are not considered 'text' within the document
as a whole.
But a user who calls script.get_text() is working from a different
definition of 'text' than a user who calls div.get_text()--otherwise
there would be no need to call script.get_text() at all. In 4.10.0,
the contents of (e.g.) a <script> tag are considered 'text' during a
get_text() call on the tag itself, but not considered 'text' during
a get_text() call on the tag's parent.
Because of this change, calling get_text() on each child of a tag
may now return a different result than calling get_text() on the tag
itself. That's because different tags now have different
understandings of what counts as 'text'. [bug=1906226] [bug=1868861]
* NavigableString and its subclasses now implement the get_text()
method, as well as the properties .strings and
.stripped_strings. These methods will either return the string
itself, or nothing, so the only reason to use this is when iterating
over a list of mixed Tag and NavigableString objects. [bug=1904309]
* The 'html5' formatter now treats attributes whose values are the
empty string as HTML boolean attributes. Previously (and in other
formatters), an attribute value must be set as None to be treated as
a boolean attribute. In a future release, I plan to also give this
behavior to the 'html' formatter. Patch by Isaac Muse. [bug=1915424]
* The 'replace_with()' method now takes a variable number of arguments,
and can be used to replace a single element with a sequence of elements.
Patch by Bill Chandos. [rev=605]
* Corrected output when the namespace prefix associated with a
namespaced attribute is the empty string, as opposed to
None. [bug=1915583]
* Performance improvement when processing tags that speeds up overall
tree construction by 2%. Patch by Morotti. [bug=1899358]
* Corrected the use of special string container classes in cases when a
single tag may contain strings with different containers; such as
the <template> tag, which may contain both TemplateString objects
and Comment objects. [bug=1913406]
* The html.parser tree builder can now handle named entities
found in the HTML5 spec in much the same way that the html5lib
tree builder does. Note that the lxml HTML tree builder doesn't handle
named entities this way. [bug=1924908]
* Added a second way to pass specify encodings to UnicodeDammit and
EncodingDetector, based on the order of precedence defined in the
HTML5 spec, starting at:
https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding
Encodings in 'known_definite_encodings' are tried first, then
byte-order-mark sniffing is run, then encodings in 'user_encodings'
are tried. The old argument, 'override_encodings', is now a
deprecated alias for 'known_definite_encodings'.
This changes the default behavior of the html.parser and lxml tree
builders, in a way that may slightly improve encoding
detection but will probably have no effect. [bug=1889014]
* Improve the warning issued when a directory name (as opposed to
the name of a regular file) is passed as markup into the BeautifulSoup
constructor. [bug=1913628]
```
### 4.9.3
```
* Implemented a significant performance optimization to the process of
searching the parse tree. Patch by Morotti. [bug=1898212]
```
### 4.9.2
```
* Fixed a bug that caused too many tags to be popped from the tag
stack during tree building, when encountering a closing tag that had
no matching opening tag. [bug=1880420]
* Fixed a bug that inconsistently moved elements over when passing
a Tag, rather than a list, into Tag.extend(). [bug=1885710]
* Specify the soupsieve dependency in a way that complies with
PEP 508. Patch by Mike Nerone. [bug=1893696]
* Change the signatures for BeautifulSoup.insert_before and insert_after
(which are not implemented) to match PageElement.insert_before and
insert_after, quieting warnings in some IDEs. [bug=1897120]
```
### 4.9.1
```
* Added a keyword argument 'on_duplicate_attribute' to the
BeautifulSoupHTMLParser constructor (used by the html.parser tree
builder) which lets you customize the handling of markup that
contains the same attribute more than once, as in:
<a href="url1" href="url2"> [bug=1878209]
* Added a distinct subclass, GuessedAtParserWarning, for the warning
issued when BeautifulSoup is instantiated without a parser being
specified. [bug=1873787]
* Added a distinct subclass, MarkupResemblesLocatorWarning, for the
warning issued when BeautifulSoup is instantiated with 'markup' that
actually seems to be a URL or the path to a file on
disk. [bug=1873787]
* The new NavigableString subclasses (Stylesheet, Script, and
TemplateString) can now be imported directly from the bs4 package.
* If you encode a document with a Python-specific encoding like
'unicode_escape', that encoding is no longer mentioned in the final
XML or HTML document. Instead, encoding information is omitted or
left blank. [bug=1874955]
* Fixed test failures when run against soupselect 2.0. Patch by Tomáš
Chvátal. [bug=1872279]
```
### 4.9.0
```
* Added PageElement.decomposed, a new property which lets you
check whether you've already called decompose() on a Tag or
NavigableString.
* Embedded CSS and Javascript is now stored in distinct Stylesheet and
Script tags, which are ignored by methods like get_text() since most
people don't consider this sort of content to be 'text'. This
feature is not supported by the html5lib treebuilder. [bug=1868861]
* Added a Russian translation by 'authoress' to the repository.
* Fixed an unhandled exception when formatting a Tag that had been
decomposed.[bug=1857767]
* Fixed a bug that happened when passing a Unicode filename containing
non-ASCII characters as markup into Beautiful Soup, on a system that
allows Unicode filenames. [bug=1866717]
* Added a performance optimization to PageElement.extract(). Patch by
Arthur Darcet.
```
### 4.8.2
```
* Added Python docstrings to all public methods of the most commonly
used classes.
* Added a Chinese translation by Deron Wang and a Brazilian Portuguese
translation by Cezar Peixeiro to the repository.
* Fixed two deprecation warnings. Patches by Colin
Watson and Nicholas Neumann. [bug=1847592] [bug=1855301]
* The html.parser tree builder now correctly handles DOCTYPEs that are
not uppercase. [bug=1848401]
* PageElement.select() now returns a ResultSet rather than a regular
list, making it consistent with methods like find_all().
```
### 4.8.1
```
* When the html.parser or html5lib parsers are in use, Beautiful Soup
will, by default, record the position in the original document where
each tag was encountered. This includes line number (Tag.sourceline)
and position within a line (Tag.sourcepos). Based on code by Chris
Mayo. [bug=1742921]
* When instantiating a BeautifulSoup object, it's now possible to
provide a dictionary ('element_classes') of the classes you'd like to be
instantiated instead of Tag, NavigableString, etc.
* Fixed the definition of the default XML namespace when using
lxml 4.4. Patch by Isaac Muse. [bug=1840141]
* Fixed a crash when pretty-printing tags that were not created
during initial parsing. [bug=1838903]
* Copying a Tag preserves information that was originally obtained from
the TreeBuilder used to build the original Tag. [bug=1838903]
* Raise an explanatory exception when the underlying parser
completely rejects the incoming markup. [bug=1838877]
* Avoid a crash when trying to detect the declared encoding of a
Unicode document. [bug=1838877]
* Avoid a crash when unpickling certain parse trees generated
using html5lib on Python 3. [bug=1843545]
```
### 4.8.0
```
This release focuses on making it easier to customize Beautiful Soup's
input mechanism (the TreeBuilder) and output mechanism (the Formatter).
* You can customize the TreeBuilder object by passing keyword
arguments into the BeautifulSoup constructor. Those keyword
arguments will be passed along into the TreeBuilder constructor.
The main reason to do this right now is to change how which
attributes are treated as multi-valued attributes (the way 'class'
is treated by default). You can do this with the
'multi_valued_attributes' argument. [bug=1832978]
* The role of Formatter objects has been greatly expanded. The Formatter
class now controls the following:
- The function to call to perform entity substitution. (This was
previously Formatter's only job.)
- Which tags should be treated as containing CDATA and have their
contents exempt from entity substitution.
- The order in which a tag's attributes are output. [bug=1812422]
- Whether or not to put a '/' inside a void element, e.g. '<br/>' vs '<br>'
All preexisting code should work as before.
* Added a new method to the API, Tag.smooth(), which consolidates
multiple adjacent NavigableString elements. [bug=1697296]
* ' (which is valid in XML, XHTML, and HTML 5, but not HTML 4) is always
recognized as a named entity and converted to a single quote. [bug=1818721]
```
Links
- PyPI: https://pypi.org/project/beautifulsoup4
- Changelog: https://data.safetycli.com/changelogs/beautifulsoup4/
This PR updates beautifulsoup4 from 4.7.1 to 4.12.3.
Changelog
### 4.11.1 ``` This release was done to ensure that the unit tests are packaged along with the released source. There are no functionality changes in this release, but there are a few other packaging changes: * The Japanese and Korean translations of the documentation are included. * The changelog is now packaged as CHANGELOG, and the license file is packaged as LICENSE. NEWS.txt and COPYING.txt are still present, but may be removed in the future. * TODO.txt is no longer packaged, since a TODO is not relevant for released code. ``` ### 4.11.0 ``` * Ported unit tests to use pytest. * Added special string classes, RubyParenthesisString and RubyTextString, to make it possible to treat ruby text specially in get_text() calls. [bug=1941980] * It's now possible to customize the way output is indented by providing a value for the 'indent' argument to the Formatter constructor. The 'indent' argument works very similarly to the argument of the same name in the Python standard library's json.dump() function. [bug=1955497] * If the charset-normalizer Python module (https://pypi.org/project/charset-normalizer/) is installed, Beautiful Soup will use it to detect the character sets of incoming documents. This is also the module used by newer versions of the Requests library. For the sake of backwards compatibility, chardet and cchardet both take precedence if installed. [bug=1955346] * Added a workaround for an lxml bug (https://bugs.launchpad.net/lxml/+bug/1948551) that causes problems when parsing a Unicode string beginning with BYTE ORDER MARK. [bug=1947768] * Issue a warning when an HTML parser is used to parse a document that looks like XML but not XHTML. [bug=1939121] * Do a better job of keeping track of namespaces as an XML document is parsed, so that CSS selectors that use namespaces will do the right thing more often. [bug=1946243] * Some time ago, the misleadingly named "text" argument to find-type methods was renamed to the more accurate "string." But this supposed "renaming" didn't make it into important places like the method signatures or the docstrings. That's corrected in this version. "text" still works, but will give a DeprecationWarning. [bug=1947038] * Fixed a crash when pickling a BeautifulSoup object that has no tree builder. [bug=1934003] * Fixed a crash when overriding multi_valued_attributes and using the html5lib parser. [bug=1948488] * Standardized the wording of the MarkupResemblesLocatorWarning warnings to omit untrusted input and make the warnings less judgmental about what you ought to be doing. [bug=1955450] * Removed support for the iconv_codec library, which doesn't seem to exist anymore and was never put up on PyPI. (The closest replacement on PyPI, iconv_codecs, is GPL-licensed, so we can't use it--it's also quite old.) ``` ### 4.10.0 ``` * This is the first release of Beautiful Soup to only support Python 3. I dropped Python 2 support to maintain support for newer versions (58 and up) of setuptools. See: https://github.com/pypa/setuptools/issues/2769 [bug=1942919] * The behavior of methods like .get_text() and .strings now differs depending on the type of tag. The change is visible with HTML tags like <script>, <style>, and <template>. Starting in 4.9.0, methods like get_text() returned no results on such tags, because the contents of those tags are not considered 'text' within the document as a whole. But a user who calls script.get_text() is working from a different definition of 'text' than a user who calls div.get_text()--otherwise there would be no need to call script.get_text() at all. In 4.10.0, the contents of (e.g.) a <script> tag are considered 'text' during a get_text() call on the tag itself, but not considered 'text' during a get_text() call on the tag's parent. Because of this change, calling get_text() on each child of a tag may now return a different result than calling get_text() on the tag itself. That's because different tags now have different understandings of what counts as 'text'. [bug=1906226] [bug=1868861] * NavigableString and its subclasses now implement the get_text() method, as well as the properties .strings and .stripped_strings. These methods will either return the string itself, or nothing, so the only reason to use this is when iterating over a list of mixed Tag and NavigableString objects. [bug=1904309] * The 'html5' formatter now treats attributes whose values are the empty string as HTML boolean attributes. Previously (and in other formatters), an attribute value must be set as None to be treated as a boolean attribute. In a future release, I plan to also give this behavior to the 'html' formatter. Patch by Isaac Muse. [bug=1915424] * The 'replace_with()' method now takes a variable number of arguments, and can be used to replace a single element with a sequence of elements. Patch by Bill Chandos. [rev=605] * Corrected output when the namespace prefix associated with a namespaced attribute is the empty string, as opposed to None. [bug=1915583] * Performance improvement when processing tags that speeds up overall tree construction by 2%. Patch by Morotti. [bug=1899358] * Corrected the use of special string container classes in cases when a single tag may contain strings with different containers; such as the <template> tag, which may contain both TemplateString objects and Comment objects. [bug=1913406] * The html.parser tree builder can now handle named entities found in the HTML5 spec in much the same way that the html5lib tree builder does. Note that the lxml HTML tree builder doesn't handle named entities this way. [bug=1924908] * Added a second way to pass specify encodings to UnicodeDammit and EncodingDetector, based on the order of precedence defined in the HTML5 spec, starting at: https://html.spec.whatwg.org/multipage/parsing.html#parsing-with-a-known-character-encoding Encodings in 'known_definite_encodings' are tried first, then byte-order-mark sniffing is run, then encodings in 'user_encodings' are tried. The old argument, 'override_encodings', is now a deprecated alias for 'known_definite_encodings'. This changes the default behavior of the html.parser and lxml tree builders, in a way that may slightly improve encoding detection but will probably have no effect. [bug=1889014] * Improve the warning issued when a directory name (as opposed to the name of a regular file) is passed as markup into the BeautifulSoup constructor. [bug=1913628] ``` ### 4.9.3 ``` * Implemented a significant performance optimization to the process of searching the parse tree. Patch by Morotti. [bug=1898212] ``` ### 4.9.2 ``` * Fixed a bug that caused too many tags to be popped from the tag stack during tree building, when encountering a closing tag that had no matching opening tag. [bug=1880420] * Fixed a bug that inconsistently moved elements over when passing a Tag, rather than a list, into Tag.extend(). [bug=1885710] * Specify the soupsieve dependency in a way that complies with PEP 508. Patch by Mike Nerone. [bug=1893696] * Change the signatures for BeautifulSoup.insert_before and insert_after (which are not implemented) to match PageElement.insert_before and insert_after, quieting warnings in some IDEs. [bug=1897120] ``` ### 4.9.1 ``` * Added a keyword argument 'on_duplicate_attribute' to the BeautifulSoupHTMLParser constructor (used by the html.parser tree builder) which lets you customize the handling of markup that contains the same attribute more than once, as in: <a href="url1" href="url2"> [bug=1878209] * Added a distinct subclass, GuessedAtParserWarning, for the warning issued when BeautifulSoup is instantiated without a parser being specified. [bug=1873787] * Added a distinct subclass, MarkupResemblesLocatorWarning, for the warning issued when BeautifulSoup is instantiated with 'markup' that actually seems to be a URL or the path to a file on disk. [bug=1873787] * The new NavigableString subclasses (Stylesheet, Script, and TemplateString) can now be imported directly from the bs4 package. * If you encode a document with a Python-specific encoding like 'unicode_escape', that encoding is no longer mentioned in the final XML or HTML document. Instead, encoding information is omitted or left blank. [bug=1874955] * Fixed test failures when run against soupselect 2.0. Patch by Tomáš Chvátal. [bug=1872279] ``` ### 4.9.0 ``` * Added PageElement.decomposed, a new property which lets you check whether you've already called decompose() on a Tag or NavigableString. * Embedded CSS and Javascript is now stored in distinct Stylesheet and Script tags, which are ignored by methods like get_text() since most people don't consider this sort of content to be 'text'. This feature is not supported by the html5lib treebuilder. [bug=1868861] * Added a Russian translation by 'authoress' to the repository. * Fixed an unhandled exception when formatting a Tag that had been decomposed.[bug=1857767] * Fixed a bug that happened when passing a Unicode filename containing non-ASCII characters as markup into Beautiful Soup, on a system that allows Unicode filenames. [bug=1866717] * Added a performance optimization to PageElement.extract(). Patch by Arthur Darcet. ``` ### 4.8.2 ``` * Added Python docstrings to all public methods of the most commonly used classes. * Added a Chinese translation by Deron Wang and a Brazilian Portuguese translation by Cezar Peixeiro to the repository. * Fixed two deprecation warnings. Patches by Colin Watson and Nicholas Neumann. [bug=1847592] [bug=1855301] * The html.parser tree builder now correctly handles DOCTYPEs that are not uppercase. [bug=1848401] * PageElement.select() now returns a ResultSet rather than a regular list, making it consistent with methods like find_all(). ``` ### 4.8.1 ``` * When the html.parser or html5lib parsers are in use, Beautiful Soup will, by default, record the position in the original document where each tag was encountered. This includes line number (Tag.sourceline) and position within a line (Tag.sourcepos). Based on code by Chris Mayo. [bug=1742921] * When instantiating a BeautifulSoup object, it's now possible to provide a dictionary ('element_classes') of the classes you'd like to be instantiated instead of Tag, NavigableString, etc. * Fixed the definition of the default XML namespace when using lxml 4.4. Patch by Isaac Muse. [bug=1840141] * Fixed a crash when pretty-printing tags that were not created during initial parsing. [bug=1838903] * Copying a Tag preserves information that was originally obtained from the TreeBuilder used to build the original Tag. [bug=1838903] * Raise an explanatory exception when the underlying parser completely rejects the incoming markup. [bug=1838877] * Avoid a crash when trying to detect the declared encoding of a Unicode document. [bug=1838877] * Avoid a crash when unpickling certain parse trees generated using html5lib on Python 3. [bug=1843545] ``` ### 4.8.0 ``` This release focuses on making it easier to customize Beautiful Soup's input mechanism (the TreeBuilder) and output mechanism (the Formatter). * You can customize the TreeBuilder object by passing keyword arguments into the BeautifulSoup constructor. Those keyword arguments will be passed along into the TreeBuilder constructor. The main reason to do this right now is to change how which attributes are treated as multi-valued attributes (the way 'class' is treated by default). You can do this with the 'multi_valued_attributes' argument. [bug=1832978] * The role of Formatter objects has been greatly expanded. The Formatter class now controls the following: - The function to call to perform entity substitution. (This was previously Formatter's only job.) - Which tags should be treated as containing CDATA and have their contents exempt from entity substitution. - The order in which a tag's attributes are output. [bug=1812422] - Whether or not to put a '/' inside a void element, e.g. '<br/>' vs '<br>' All preexisting code should work as before. * Added a new method to the API, Tag.smooth(), which consolidates multiple adjacent NavigableString elements. [bug=1697296] * ' (which is valid in XML, XHTML, and HTML 5, but not HTML 4) is always recognized as a named entity and converted to a single quote. [bug=1818721] ```Links
- PyPI: https://pypi.org/project/beautifulsoup4 - Changelog: https://data.safetycli.com/changelogs/beautifulsoup4/