Closed ralyodio closed 2 years ago
Off-hand that sounds like an issue with deno-dom
, given that it works if you just run this in the console in a web browser after pasting in all of Readability.
This is likely going to need more detailed steps to reproduce in order to be actionable (ie from scratch, how are you setting this up; how are you fetching the document, how are you parsing it, and even where you're located might matter because the BBC makes pretty significant changes to their site depending on where you are, probably based on geoip). A minimal testcase would be even better.
import Readability from "https://cdn.esm.sh/v57/moz-readability@0.2.1/es2021/moz-readability.js"
import { DOMParser } from "https://raw.githubusercontent.com/b-fuze/deno-dom/master/deno-dom-wasm.ts"
const url = 'https://www.bbc.com/news/science-environment-59210395';
console.log(url);
const res = await fetch(url, {
headers: {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0'
}
});
if (!res.ok) {
console.error(await res.text());
}
const body = await res.text();
const doc = new DOMParser().parseFromString(body, "text/html");
let article;
try {
console.log('parsing article');
//article = new Readability(doc.window.document).parse();
article = new Readability(doc).parse();
console.log('articled parsed');
console.log(article);
} catch(err) {
console.error(err);
}
console.log(url, body, article);
$ deno run --allow-all --unstable app.ts
I’m hitting it from USA
this page gets the content, but the cookie gdpr crap is the only thing that gets parsed.
https://www.ft.com/content/2bbedfde-c9da-4bc6-b2e6-af0698240851
short of handling each site, is there a way to just remove the top layer div that is z-index highest from the dom?
this might get rid of GDPR garbage divs
by the way, deno-dom fixed their issue that was causing this library to hang. So that's fixed. Just need a way to remove overlays for GDPR and other crap.
I found this code here From a google chrome extension which basically adds custom css to hide cookie banners:
var html = [
// This is causing two scrollbars on some sites. Need to remove and find another way to re-enable scrolling with full-page modals.
"body, html { overflow: unset !important }",
"div[id *= 'cookie'] { display: none !important }",
"div[id *= 'Cookie'] { display: none !important }",
"div[class *= 'Cookie'] { display: none !important }",
"div[class *= 'cookie'] { display: none !important }",
"div[id *= 'CCPA'] { display: none !important }",
"div[class *= 'CCPA'] { display: none !important }",
".preloaded_lightbox {display: none !important}",
".omaha-background {display: none !important}",
"#hs-eu-cookie-confirmation { display: none !important }",
"#onesignal-popover-container { display: none !important }",
".c-banner-advert-sticky { display: none !important }",
"w-div { display: none !important }",
".mfg-bg.mfp-ready { display: none !important }",
".mfg-wrap.mfp-ready { display: none !important }",
".ig_action_bar.ig_container { display: none !important }",
"div[role='dialog'][aria-label='cookieconsent'].cc-window { display: none !important }",
".mkt-at-toaster { display: none !important }",
".subscription-toaster--hidden, .subscription-toaster { display: none !important }",
".bbccookies-banner.orb-banner-wrapper.bbccookies-d { display: none !important }",
".md-modal2.flex { display: none !important }",
".modal-backdrop.in { display: none !important }",
".modal.email-submission-modal { display: none !important }",
"#rt-roadblock-modal.rt-roadblock-modal modal-dialog in { display: none !important }",
"#webmdHoverOverlay, #webmdHoverWrapper { display: none !important }",
".alert.ad-blocker { display: none !important }",
"#signup-bar.fixed.bottom { display: none !important }",
".pushcrew-chrome-style-notification, .pushcrew-chrome-style-notification-safari { display: none !important }",
".HB-Slider.hb-animated { display: none !important }",
".mcwidget-overlay { display: none !important }",
"section.newsletter-signup { display: none !important }",
"div#intentPreview { display: none !important }",
"#intentOpacityDiv { display: none !important }",
"#wzrk_wrapper { display: none !important }",
"#quick-signup { display: none !important }",
".articles-navigation { display: block !important }",
".klaviyo-form { display: none !important }",
"section#welcome-modal.welcome-modal { display: none !important }",
"cnx.cnx-video-container { display: none !important }", // autoplaying video on washingtonexaminer.com
".push-body.nao-assinante { display: none !important }", // https://www.estadao.com.br/
".ablock { display: none !important }", // https://www.phoenixnewtimes.com/news/uber-car-in-autonomous-mode-when-it-killed-woman-police-say-10247590
".all-body { filter: initial !important }", // https://www.phoenixnewtimes.com/news/uber-car-in-autonomous-mode-when-it-killed-woman-police-say-10247590
".a-modal-scroller.a-declarative { filter: initial !important }", // amazon full-page modal "Donate to smile charity"
"div[data-video-player=container] { display: none !important }", // autoplaying video on cnet.com
".fb_lightbox-overlay { display: none !important }",
"#provely-widget { display: none !important }", // https://www.toonly.com/
"div.profile-top-bar-container-searchForm { display: none !important }", // mylife.com
"div.relatedBook { display: none !important }", // http://www.informit.com/articles/article.aspx?p=18225&seqNum=2
"div.-gtqce-onaare { display: none !important }", // sportingnews.com cookie banner
"iframe[allow='autoplay; fullscreen'][x-enc='src'] { display: none !important }", // sportingnews.com <-- autoplaying video
"div.s2nPlayerFrame { display: none !important }", // https://smallbusiness.chron.com/open-new-tab-clicking-google-chrome-63612.html
"iframe#um_ultimedia_wrapper_iframeUltimedia { display: none !important }", // denofgeek.com autoplaying video
"div.sticky-video { display: none !important }", // https://www.independent.co.uk/news/science/biological-clock-ageing-turn-back-reverse-study-new-a9094261.html
".f-NJD1R8-1- { display: none !important }",
".tve-leads-lightbox { display: none !important }", // mouse out of page: https://www.rubyguides.com/2019/07/ruby-string-concatenation/
".mailmunch-popover, mailmunch-overlay { display: none !important }", // mouse out of page: http://inmyownterms.com/take-note-languages-codes-versus-country-codes/
".email-capture-popup-wrapper { display: none !important }", // email signup after 1 minute: https://www.hunker.com/13400974/how-much-do-bath-fitters-cost
".jwplayer__container--sticky { display: none !important }", // hunker.com -- self-playing video
"#emailSignUpDialog { display: none !important }", // lowes.com newsletter signup
".pum-overlay { display: none !important }", // https://daily.jstor.org/the-rise-and-fall-of-the-pet-bird/
"div[class *= 'sp_message_'], div[id *= 'sp_message_'] { display: none !important }", // denofgeek.com/uk
"div[class *= 'sp_veil'] { display: none !important }", // denofgeek.com/uk
"div#iubenda-cs-banner { display: none !important }", // http://www.finsmes.com/2019/10/edify-labs-secures-10m-in-seed-funding.html
".boxzilla-container { display: none !important }", // https://thebaffler.com/latest/corrupted-headspace-semley
"._evidon_banner { display: none !important }", // https://www.1843magazine.com/food-drink/ikejime-a-humane-way-to-kill-fish-that-makes-them-tastier
"div#paywall-banner { display: none !important }", // https://www.bloomberg.com/quote/APPL:US
".sa-soft-rb-v1 { display: none !important }", // https://seekingalpha.com/news/3505376-pg-and-e-rejects-san-franciscos-2_5b-asset-purchase-offer
"#CybotCookiebotDialog { display: none !important }", // https://www.fanatical.com/en/
".a-declarative { display: none !important }", // amazon popup
".login-modal-div { display: none !important }", // https://www.geeksforgeeks.org/css-attribute-selector/
".ModalContainer_widget_bannerVisible { display: none !important }", // https://www.teslarati.com/spacex-prepared-first-dual-falcon-9-fairing-catch/
"div#email-capture { display: none !important }", // https://www.techwalla.com/articles/how-to-set-up-wireless-printing-with-a-samsung-clp-315w
"div[data-hypernova-key='popup_block'] { display: none !important }", // wayfair.com
"div.exit_popup { display: none !important }", // when mouse leaves page, newsletter signup popup: https://severalnines.com/blog/how-perform-online-schema-changes-mysql-using-gh-ost
"div#consent_blackbar { display: none !important }", // https://www.redhat.com/en/about/press-releases/ibm-closes-landmark-acquisition-red-hat-34-billion-defines-open-hybrid-cloud-future
"div#sitewide-lightbox-container { display: none !important }", // newsletter signup overstock.com
"div.Campaign.CampaignType--popup { display: none !important }", // mouseout popup https://www.guru99.com/oltp-vs-olap.html
"div#adtoniq-msgr-bar { display: none !important }", // https://dzone.com/articles/what-is-viewencapsulation-in-angular
"div.fancybox-overlay { display: none !important }", // macworld.com fullscreen popup
".fancybox-lock { overflow: auto !important }", // overwrite CSS that turns off scrolling for FancyBox
".met-flyout { display: none !important }", // latimes.com
"div[id *= 'simplemodal'], div.simplemodal-container { display: none !important }", // developers.meethue.com newsletter signup modal
".mpp-container { display: none !important }", // roadtovr.com newsletter signup
"div#WhitelistOverlayModalBackground { display: none !important }", // kbb.com ad blocker modal
"div.wrap div.mh-message-bar { display: none !important }", // spotify.com/it cookie banner
"div.klaviyo-form.needsclick { display: none !important }", // zocalofoods.com/cart newsletter popup
"div.dailymotion-cpe { display: none !important }", // video autoplayer on chichester.co.uk/news
"iframe.syndicated-modal { display: none !important }", // https://www.propublica.org/article/the-irs-decided-to-get-tough-against-microsoft-microsoft-got-tougher
".nytc---modal-window---windowContainer { display: none !important }", // https://cooking.nytimes.com/recipes/9465-the-250-cookie-recipe
"#_evidon-barrier-wrapper { display: none !important }", // cookie banner https://www.cosmopolitan.com/uk/reports/a32050083/rubber-gloves-coronavirus-spreading-tiktok/
".tp-modal, .tp-backdrop.tp-active { display: none !important }", // account signup thedailybest.com
".tp-modal-open { overflow: auto !important }", // turn on the page scrolling for TP-Modal pages (thedailybest.com)
".jumpstart-sticky-active { position: static !important }", // video autoplayer https://www.myrecipes.com/recipe/baked-italian-style-cauliflower
".cc_banner.cc_container { display: none !important }", // cookiie banner https://thelittlepotcompany.co.uk/blogs/pottery/making-your-own-pottery-glaze
".fc-ab-root { display: none !important }", // disable ad blocker http://www.foxnews.com
".sok-browser-consent-modal { display: none !important }", // cookie banner https://yhteishyva.fi/reseptit/suklaa-mascarponekakku/recipe-5482
"#onetrust-consent-sdk { display: none !important }", // privacy popup modal https://www.cnn.com
"#ins-frameless-overlay, .ins-preview-wrapper { display: none !important }", // full page modal to turn on notifications? super stupid https://english.alarabiya.net/en/features/2019/10/02/Harassment-and-imprisonment-Life-as-persecuted-Christian-in-Iran.html
".soundest-form-image-left-overlay, .soundest-form-image-left { display: none !important }", // newsletter signup https://promarinesupplies.com/blogs/blog/tips-for-tinting-epoxy-resin
"#consent-banner { display: none !important } ", // cookie banner https://www.meillakotona.fi/reseptit/meheva-suklaakakku
".modal-overlay.nlPopup.active { display: none !important }", // delayed newsletter signup timesofisrael.com
"#abd-banner { display: none !important }", // disable ad blocker macrotrends.net
".leadinModal { display: none !important }", // mouseout banner https://blog.hubspot.com/marketing/best-email-subject-lines-list
".seo_lightbox.entered.user_dismiss { display: none !important }", // mouseout banner scribd.com
".periodic-modal-popup { display: none !important }", // newsletter signup adoptapet.com
"#mailing-list-popup { display: none !important }", // newsletter signup https://jacobinmag.com/2020/04/ecuador-lenin-moreno-coronavirus-rafael-correa-health-care
"#mailchimp-top-bar { display: none !important }", // newsletter signup http://www.finsmes.com/2019/10/edify-labs-secures-10m-in-seed-funding.html
"div.mfp-bg { display: none !important }", // https://www.zyciepabianic.pl/styl-zycia/jak-dlugo-idzie-przesylka-z-aliexpress.html
"aside#cookie-warn { display: none !important }", // https://www.zyciepabianic.pl/styl-zycia/jak-dlugo-idzie-przesylka-z-aliexpress.html
"div#tc_priv_CustomOverlay { display: none !important }", // cookie banner https://ovh.co.uk
".ctct-popup-wrapper { display: none !important }", // newsletter signup http://www.lyndhurst-oh.com/departments.html
".c-article-meter-banner { display: none !important }", // signup banner https://thespec.com
"#webmdHoverOverlay { display: none !important }", // newsletter signup https://webmd.com
"ul.notices.notices--bottom_fixer.js-notices { display: none !important }", // cookie banner https://forums.terraria.org/index.php
".cookie-bar { display: none !important }", // cookie banner https://www.flashpoint-intel.com/blog/iraq-threat-update-june-2020/
".fbs-auth__adblock { display: none !important }", // adblock banner https://www.forbes.com/sites/victoriaforster/2020/05/12/wearing-a-mask-to-reduce-the-spread-of-coronavirus-will-not-give-you-carbon-dioxide-poisoning/#ee133e517f56
"#cookiePopup + div.modal-backdrop { display: none !important }", // get rid of dark background behind cookie window https://www.t-mobile.nl/
"section#ccc[slider-optin=''] { display: none !important }", // cookie notification https://www.wien.info/de/sightseeing/architektur-design
".svs-popup-root { display: none !important }", // newsletter signup https://www.iwillteachyoualanguage.com/learn/french/french-tips/french-punctuation
".sumome-react-wysiwyg-component { display: none !important }", // newsletter signup https://www.iwillteachyoualanguage.com/learn/french/french-tips/french-punctuation
"section#ensModalWrapper { display: none !important }", // cookie banner on https://www.sherwin-williams.com
"#ccpaCookieBanner { display: none !important }", // cookie banner https://www.paypal.com
".formkit-overlay { display: none !important }", // newsletter signup https://alldayidreamaboutfood.com
".ab-in-app-message { display: none !important }", // https://www.kcra.com/article/gov-newsom-coronavirus-update-nov-16/34690298
"#PopupSignupForm_0 .mc-modal { display: none !important }", // signup form https://mobiledevmemo.com/apple-arcade-one-year-on-no-killer-games-cant-compete-with-free/
"div#view-offer.view-offer, div.piano-fixed-footer-two { display: none !important }", // signup form https://www.washingtontimes.com/news/2020/dec/1/unraveling-another-democratic-fantasy/
"div[id *= 'sp_message_container'] { display: none !important }", // a number of sites use this for cookie banners
".sp-message-open { position: initial !important; overflow: initial !important }", // goes with sp_message
"div.navi-push-notification-prompt { display: none !important }", // notification banner https://www.bloomberg.com/quote/IAG:LN
"div.cmpboxBG { display: none !important }", // cookie banner https://www.werstreamt.es/
"div[id*='gdpr'] { display: none !important }",
"div.truste_overlay, div.truste_box_overlay { display: none !important }", // cookie banner https://www.eurofinsgenomics.eu/en/ecom/checkout/login-register/
".gTMtLb > div[class=''], .m114nf.aID8W, #cnsw { display: none !important }", // "before you continue" popup
"#usercentrics-root { display: none !important }", // cookie banner https://www.ka-news.de/region/karlsruhe/Karlsruhe~/spinnenlaeufer-kalikokrebs-gottesanbeterin-diese-exoten-leben-in-karlsruhe;art6066,2587181
"div#didomi-custom-host { display: none !important }", // cookie banner https://www.hilti.si/
"#udtCookiebox, #udtDark, #udtWhite { display: none !important }", // cookie banner https://www.wolterskluwer.com/nl-nl/solutions/basecone
"cmp-banner { display: none !important }", // cookie banner https://www.sat1.de/
"div[id*='sp_veil'] { display: none !important }", // sp_veil cookie banners
"div[id*='c-dialog'] { display: none !important }", // notification banner https://welt.de
"div[class*='style__modal___'] { display: none !important }", // newsletter https://store.dji.com
"#locked-screen-notifications { display: none !important }", // https://www.24symbols.com/author/derek-prince?id=19499&locale=es
".widget-overlay-mask { display: none !important }", // background overlay https://hotels.com
".email-overlay, .overlay-wrapper { display: none !important }", // mouseout banner https://www.howtogeek.com/437624/how-to-enable-google-chromes-new-extensions-menu/
"div[data-tracking-opt-in-overlay='true'] { display: none !important }", // cookie banner https://pathofexile.gamepedia.com/Path_of_Exile_Wiki
".web-snackbar__surface { display: none !important }", // cookie banner https://web.dev
"#didomi-host::before { display: none !important }", // cookie banner background https://www.filmweb.pl
"body.preventScroll { position: initial !important }", // https://www.filmweb.pl
"#consent { display: none !important }", // cookie baner https://www.nperf.com
".cookie-notice { display: none !important }", // cookie banner https://tesa.com
".-locked body:before { position: initial !important } .-locked body:after { opacity: 1 !important; position: initial !important; background-color: initial !important }", // cookie banner https://tesa.com
"#read-overlay-container { display: none !important }", // cookie banner background https://www.saechsische.de/
"#rgpd-banner { display: none !important }", // cookie banner https://www.doktorwhatson.tv/
"div[data-testid='cookie-policy-banner'] { display: none !important }", // cookie banner https://developers.facebook.com/docs/instagram-basic-display-api/
"#_evh-ric { display: none !important }", // cookie banner https://www.la7.it/rivedila7
"div[aria-label *= 'Cookie'], div[aria-label *= 'cookie'] { display: none !important }", // https://www.metoffice.gov.uk/about-us/press-office/news/weather-and-climate/2020/storm-bella-has-been-named
"#piano_wrapper_unten { display: none !important }", // popup banner https://www.augsburger-allgemeine.de/panorama/Wann-ist-Ostern-2021-Termine-fuer-Ostersonntag-Ostermontag-und-Osterferien-id53722456.html
".privy-popup-content-wrap { display: none !important }", // newsletter signup https://oceansalive.co.uk/
".modal-backdrop { display: none !important }", // background overlay https://www.mantel.com
"#tsj-bottom-ad { display: none !important }", // https://www.thesleepjudge.com/how-much-is-a-sleep-number-bed/
];
he's basiclly just adding a cs sfile in the dom with these matches.
i used crx reader to view source of chrome extnesion "remove cookie banners"
SO I found something easier....maybe add a flag or something to removeCookieBanners: true
all we need to do is run this:
[...document.querySelectorAll('[class*="cookie" i]')].map(n => n.remove());
[...document.querySelectorAll('[id*="cookie" i]')].map(n => n.remove());
that's going to delete almost everything that guy has in his extension.
I think that you're using a very old version of Readability? Not entirely sure, but you can import version v0.4.1
with the following
import { Readability } from "https://esm.sh/@mozilla/readability@0.4.1?no-check";
vs what seems to be a much older build of v0.2.1
?
import Readability from "https://cdn.esm.sh/v57/moz-readability@0.2.1/es2021/moz-readability.js";
Which was last updated 2 years ago on NPM, vs 10 months ago for v0.4.1
?
still hangs on a lot of doms from deno-dom parser.
@ralyodio run this exact code and tell me if you still get the same problem
import { Readability } from "https://esm.sh/@mozilla/readability@0.4.1?no-check";
import { DOMParser } from "https://deno.land/x/deno_dom@v0.1.18-alpha/deno-dom-wasm.ts";
const url =
"https://www.bloomberg.com/news/articles/2021-11-13/apple-to-pay-30-million-over-security-checks-for-store-workers";
console.log(url);
const res = await fetch(url, {
headers: {
"User-Agent":
"Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0",
},
});
if (!res.ok) {
console.error(await res.text());
}
const body = await res.text();
const doc = new DOMParser().parseFromString(body, "text/html");
console.log("parsing article");
const article = new Readability(doc).parse();
console.log("articled parsed");
console.log(article);
still hangs on a lot of doms from deno-dom parser.
At this point I think this is a deno-dom issue, and shouldn't be triaged further here, given the earlier comment where the deno-dom folks fixed something themselves.
Yeah that code works now.
still hangs with puppetter page content though:
this body causes parser to hang
import { Readability } from "https://esm.sh/@mozilla/readability@0.4.1?no-check";
import { DOMParser } from "https://deno.land/x/deno_dom@v0.1.18-alpha/deno-dom-wasm.ts";
const res = await fetch('http://sprunge.us/lS60yJ');
const body = await res.text();
const doc = new DOMParser().parseFromString(body, 'text/html');
console.log('parsing article');
const article = new Readability(doc).parse();
console.log('articled parsed!');
console.log(article);
still hangs with puppetter page content though:
this body causes parser to hang
This is running into the issue fixed in https://github.com/mozilla/readability/pull/694 , because apparently deno-dom also doesn't think that Node.childNodes
should be a live array. It gets stuck trying to move all the children out of a DOM node because the DOM implementation lies and says the children are still there. That's really a bug in the DOM library, but anyway - if you use current git tip the problem goes away. We haven't done a point release with this fix yet.
If you still have more issues, please do more debugging - given it hangs in a single while
loop, if you use --inspect
with deno
and then use the devtools in a chromium browser and attach, you can just break execution and see what's happening.
Trying to parse the body from a bbc link:
https://bbc.co.uk
It never reaches line 41 with bbc html
I'm using deno-dom in deno