Open shtse8 opened 7 years ago
I have tried DomCrawler, but it is fine without any problem when encoding Utf-8.
Code:
use Symfony\Component\DomCrawler\Crawler;
$html = <<<EOF
<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!</title>
<body>
网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!
</body>
</html>
EOF;
$crawler = new Crawler($html);
echo $crawler->html();
Result:
<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!</title>
</head>
<body>
网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!
</body>
And I find they have fixed this bug already. https://github.com/symfony/dom-crawler/pull/4
I am trying to fix this problem. https://github.com/wasinger/htmlpagedom/pull/19
I think it can help you curl_setopt($ch, CURLOPT_ENCODING, 'UTF-8');
Any update on this issue, do we still have problem with UTF-8 because this is a huge problem if it does exist, most sites use UTF-8 anyhow.
Same problem here, in v1.3.
I just decode entities after the save:
$html = html_entity_decode($html, ENT_NOQUOTES, 'UTF-8');
seems the underlying problem is that symfony/dom-crawler switches to entities to avoid some other bugs:
I just decode entities after the save:
$html = html_entity_decode($html, ENT_NOQUOTES, 'UTF-8');
I think this is not good idea - because this decode all entities from HTML (for example i have bigger document where can by for example used > or ).
I found solution which work for me - in line:
https://github.com/wasinger/htmlpagedom/blob/563bc7a399b473631b644cc35e63202fd61987ac/src/HtmlPageCrawler.php#L887 i change from:
return $this->getDOMDocument()->saveHTML();
to:
return $this->getDOMDocument()->saveHTML($this->getDOMDocument()->documentElement);
based by https://stackoverflow.com/a/20675396
But my solution for some reason remove DOCTYPE in this test script:
<?php
$html = <<<EOF
<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#">
<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!</title>
</head>
<body>
网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!
</body>
</html>
EOF;
use \Wa72\HtmlPageDom\HtmlPageCrawler;
$document = new HtmlPageCrawler($html);
echo "--- HtmlPageDom -----------------------------------------------------------" .PHP_EOL.PHP_EOL;
echo $document->saveHTML();
echo PHP_EOL;
use Symfony\Component\DomCrawler\Crawler;
$crawler = new Crawler($html);
echo "--- DomCrawler ------------------------------------------------------------" .PHP_EOL.PHP_EOL;
echo $crawler->html();
This is output:
vagrant@d8:/data/uniweb/uniweb-cms/cms3[feature/UCMS-313-content-migration-blogs *]$ drush @lv scr test.php
--- HtmlPageDom -----------------------------------------------------------
<html prefix="og: http://ogp.me/ns#">
<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!</title>
</head>
<body>
网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!
</body>
</html>
--- DomCrawler ------------------------------------------------------------
<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!</title>
</head>
<body>
网友终于肉搜出「范冰冰」家族照片,没想到看见她奶奶才发现「范冰冰是全家最难看的」!
</body>
html_entity_decode
is perfectly valid, if you wanted to have putput as &
then it should be in source document as &amp;
.
I make small test script for compare various DOM parsers - https://github.com/havran/php-html-parsers-test
Old simplehtmldom seems still best :-).
Code to reproduce:
Result:
Expected Result:
It is a known bug of PHP DomDocument. Here is the reference: http://stackoverflow.com/questions/8218230/php-domdocument-loadhtml-not-encoding-utf-8-correctly