Open kvetoslavnovak opened 11 months ago
OK. This seemt to do the trick:
cheerio(elem).html().replaceAll(/<\/?[a-zA-Z0-9=" ]*>/g, ' ').replace(/\s\s+/g, ' ').trim())
But better would if cheerio keep the whitespace which can ve trime manually as user needs to.
Facing the same issue and hitting it via langchain CheerioWebLoad so I cant use the approach above.
I believe this is related https://github.com/cheeriojs/cheerio/issues/2841 and really unfortunate 😬 That regex can get heavy for huge htmls
text() method in JQuery keeps whitespaces between text parts. Cheerio unfortunatelly merges text parts from tags thogether givign you quite gibberish words.
See exampel https://api.jquery.com/text/ I tested it by myself and JQuery text() method keeps whitespaces between the text parts of the tags. This is very useful.
Cheerio on the other hand messes text parts alltogther without no spaces which is a pain.
E.g. from this
JQuery text() gives you expected text
PART ONE GENERAL PROVISIONS
but Cheerio gives youPART ONEGENERAL PROVISIONS
Also standard HTML innerText property works the same.