GeneThomas
commented
4 years ago Small tweak to text extraction code to match web browsers’ “one <br> two” -> “one\ntwo”
static string ExtractText(IElement elem)
{
//elem.Text;
//elem.TextContent
string str = elem.InnerHtml;
str = Regex.Replace(str, "\\s", " ");
str = str.Replace(" ", " ");
str = str.Replace("<br>", "\n");
str = Regex.Replace(str, "<[^>]+>", ""); // remove elements
str = Regex.Replace(str, " +", " "); // many spaces are one space
str = Regex.Replace(str, " *\n *", "\n"); // “one <br> two” -> “one\ntwo”
str = str.Replace(">", ">");
str = str.Replace("<", "<");
str = str.Replace(""", "\"");
str = str.Replace("'", "'");
str = str.Replace("&", "&");
// any other html &entity; possible :o(
str = str.Trim(' '); // leave leading and trailsing \n (<br>)s
return str;
}
FlorianRappl
New Feature Proposal
When extracting information from web pages, such as the main table #main_table_countries_today in https://www.worldometers.info/coronavirus/ one wants the text as the user sees it: \n and \t converted to space,
as \n; html entities expanded, e.g. > → >; multiple spaces to one space and text Trim().
Presently the .Text() and .TextContent of the second header cells are “TotalCases” as the <br> has been removed. Implementing this on top of Angle Sharp is not ideal as one has to implement all of the html entities such as $quot.
If you do not want to break existing users of .Text [you could since you have not got to v1.0 yet] a property called .UserText would be good.
The following code does most of that that I requrie:
Attached is a Visual Studio C# project that shows the feature, the user sees:
but .Text() and .TextContent return “onetwo”:
AngleSharpBrFault.zip .