Puppeteer wacht mogelijk niet lang genoeg op brakke websites

[simonbesters]

https://www.vuuniversitypress.com laad op een vreemde manier in twee stappen, waardoor wappa denkt dat de site al klaar is terwijl er nog niks gebeurd is.

Je kunt Puppeteer laten wachten tot de timeout (30s meestal), tot een page load (dat lijkt te gebeuren op de vuu site) of bijvoorbeeld tot networkidle0 of networkidle2. Zie:

https://pptr.dev/api/puppeteer.puppeteerlifecycleevent

In driver.js op regel 750 zou deze aanpassing kunnen helpen dit op te lossen en ook te voorkomen dat wappa niet 30s hoeft te wachten op extreem trage websites.

Nog niet onderzocht wat dit voor verschillende scenario's betekent.

try {
          await page.goto(url.href, {
              waitUntil: 'networkidle0'
          })

[rudiedirkx]

Als ik lokaal https://www.vuuniversitypress.com/ doe, vindt ie 50% Laravel en helemaal geen Wordpress.

waitUntil verandert niks. Heel veel cookies, maar geen CMS-achtige. Wappa doet al een 3 sec sleep (wat ik blijkbaar overal in elke issue noem tegenwoordig 😆), zou die waitUntil dan nog veel doen? Langere pause helpt ook niet.

[rudiedirkx]

Volgens welke patterns zou Wappa Wordpress moeten vinden vind jij?

[simonbesters]

Wordpress, Woocommerce en een hele rimbam aan plugins/zooi van WP. En in het verleden vonden we die ook met wappa:

https://app.digitalinsightsplatform.nl/sites/7504/check-history/4?hilite=1769106

[simonbesters]

Met de voorgestelde fix (await sleep(1000) in #1 krijg je nu wel resultaat:

vuuniversitypress.com met sleep

``` wappalyzer % node cli.js 'https://vuuniversitypress.com' -e -d --pretty log | driver | Launching browser (attempt 1)... log | driver | Clearing cookies... log | driver | Setting storage... log | driver | Navigate to https://vuuniversitypress.com/ log | driver | Adding 1 detections (headers, certIssuer) log | driver | Adding 0 detections (xhr) log | driver | Sleeping for 3000 ms (after chromium.goto) log | driver | Adding 134 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 31 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 0 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Adding 1 detections (dom, js, url, cookies, cookieNames, html, text, css, scripts, scriptSrc, meta) log | driver | Page closed (https://vuuniversitypress.com/) { "urls": { "https://vuuniversitypress.com/": { "status": 200 } }, "technologies": [ { "slug": "wordpress", "name": "WordPress", "description": "WordPress is a free and open-source content management system written in PHP and paired with a MySQL or MariaDB database. Features include a plugin architecture and a template system.", "confidence": 100, "version": "6.4.3", "icon": "WordPress.svg", "website": "https://wordpress.org", "cpe": "cpe:2.3:a:wordpress:wordpress:*:*:*:*:*:*:*:*", "categories": [ { "id": 1, "slug": "cms", "name": "CMS" }, { "id": 11, "slug": "blogs", "name": "Blogs" } ], "rootPath": true }, { "slug": "cart-functionality", "name": "Cart Functionality", "description": "Websites that have a shopping cart or checkout page, either using a known ecommerce platform or a custom solution.", "confidence": 100, "version": null, "icon": "Cart-generic.svg", "website": "https://www.wappalyzer.com/technologies/ecommerce/cart-functionality", "cpe": null, "categories": [ { "id": 6, "slug": "ecommerce", "name": "Ecommerce" } ], "rootPath": true }, { "slug": "mysql", "name": "MySQL", "description": "MySQL is an open-source relational database management system.", "confidence": 100, "version": null, "icon": "MySQL.svg", "website": "https://mysql.com", "cpe": "cpe:2.3:a:mysql:mysql:*:*:*:*:*:*:*:*", "categories": [ { "id": 34, "slug": "databases", "name": "Databases" } ] }, { "slug": "php", "name": "PHP", "description": "PHP is a general-purpose scripting language used for web development.", "confidence": 100, "version": null, "icon": "PHP.svg", "website": "https://php.net", "cpe": "cpe:2.3:a:php:php:*:*:*:*:*:*:*:*", "categories": [ { "id": 27, "slug": "programming-languages", "name": "Programming languages" } ] }, { "slug": "google-sign-in", "name": "Google Sign-in", "description": "Google Sign-In is a secure authentication system that reduces the burden of login for users, by enabling them to sign in with their Google account.", "confidence": 100, "version": null, "icon": "Google.svg", "website": "https://developers.google.com/identity/sign-in/web", "cpe": null, "categories": [ { "id": 69, "slug": "authentication", "name": "Authentication" } ], "rootPath": true }, { "slug": "vimeo", "name": "Vimeo", "description": "Vimeo is a video hosting, sharing and services platform. Vimeo operation an ad-free basis by providing subscription plans.", "confidence": 100, "version": null, "icon": "Vimeo.svg", "website": "https://vimeo.com", "cpe": null, "categories": [ { "id": 14, "slug": "video-players", "name": "Video players" } ], "rootPath": true }, { "slug": "bootstrap", "name": "Bootstrap", "description": "Bootstrap is a free and open-source CSS framework directed at responsive, mobile-first front-end web development. It contains CSS and JavaScript-based design templates for typography, forms, buttons, navigation, and other interface components.", "confidence": 100, "version": "3.3.7", "icon": "Bootstrap.svg", "website": "https://getbootstrap.com", "cpe": "cpe:2.3:a:getbootstrap:bootstrap:*:*:*:*:*:*:*:*", "categories": [ { "id": 66, "slug": "ui-frameworks", "name": "UI frameworks" } ], "rootPath": true }, { "slug": "elementor", "name": "Elementor", "description": "Elementor is a website builder platform for professionals on WordPress.", "confidence": 100, "version": "3.18.3", "icon": "Elementor.svg", "website": "https://elementor.com", "cpe": null, "categories": [ { "id": 51, "slug": "page-builders", "name": "Page builders" }, { "id": 87, "slug": "wordpress-plugins", "name": "WordPress plugins" } ], "rootPath": true }, { "slug": "woocommerce", "name": "WooCommerce", "description": "WooCommerce is an open-source ecommerce plugin for WordPress.", "confidence": 100, "version": "8.5.1", "icon": "WooCommerce.svg", "website": "https://woocommerce.com", "cpe": null, "categories": [ { "id": 6, "slug": "ecommerce", "name": "Ecommerce" }, { "id": 87, "slug": "wordpress-plugins", "name": "WordPress plugins" } ], "rootPath": true }, { "slug": "react", "name": "React", "description": "React is an open-source JavaScript library for building user interfaces or UI components.", "confidence": 100, "version": "18.2.0", "icon": "React.svg", "website": "https://reactjs.org", "cpe": "cpe:2.3:a:facebook:react:*:*:*:*:*:*:*:*", "categories": [ { "id": 12, "slug": "javascript-frameworks", "name": "JavaScript frameworks" } ], "rootPath": true }, { "slug": "gsap", "name": "GSAP", "description": "GSAP is an animation library that allows you to create animations with JavaScript.", "confidence": 100, "version": null, "icon": "TweenMax.png", "website": "https://greensock.com/gsap", "cpe": null, "categories": [ { "id": 12, "slug": "javascript-frameworks", "name": "JavaScript frameworks" } ], "rootPath": true }, { "slug": "yoast-seo", "name": "Yoast SEO", "description": "Yoast SEO is a search engine optimisation plugin for WordPress and other platforms.", "confidence": 100, "version": "21.8", "icon": "Yoast SEO.png", "website": "https://yoast.com/wordpress/plugins/seo/", "cpe": null, "categories": [ { "id": 54, "slug": "seo", "name": "SEO" }, { "id": 87, "slug": "wordpress-plugins", "name": "WordPress plugins" } ], "rootPath": true }, { "slug": "nginx", "name": "Nginx", "description": "Nginx is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache.", "confidence": 100, "version": null, "icon": "Nginx.svg", "website": "https://nginx.org/en", "cpe": "cpe:2.3:a:f5:nginx:*:*:*:*:*:*:*:*", "categories": [ { "id": 22, "slug": "web-servers", "name": "Web servers" }, { "id": 64, "slug": "reverse-proxies", "name": "Reverse proxies" } ], "rootPath": true }, { "slug": "cloudflare", "name": "Cloudflare", "description": "Cloudflare is a web-infrastructure and website-security company, providing content-delivery-network services, DDoS mitigation, Internet security, and distributed domain-name-server services.", "confidence": 100, "version": null, "icon": "CloudFlare.svg", "website": "https://www.cloudflare.com", "cpe": null, "categories": [ { "id": 31, "slug": "cdn", "name": "CDN" } ] }, { "slug": "woocommerce-multilingual", "name": "WooCommerce Multilingual", "description": "WooCommerce Multilingual plugin makes it possible to run fully multilingual ecommerce sites using WooCommerce and WPML.", "confidence": 100, "version": "5.3.2", "icon": "WooCommerce Multilingual.png", "website": "https://wordpress.org/plugins/woocommerce-multilingual", "cpe": null, "categories": [ { "id": 87, "slug": "wordpress-plugins", "name": "WordPress plugins" }, { "id": 89, "slug": "translation", "name": "Translation" } ], "rootPath": true }, { "slug": "wpml", "name": "WPML", "description": "WPML plugin makes it possible to build and run fully multilingual WordPress sites.", "confidence": 100, "version": "4.6.9", "icon": "WPML.svg", "website": "https://wpml.org/", "cpe": null, "categories": [ { "id": 87, "slug": "wordpress-plugins", "name": "WordPress plugins" }, { "id": 89, "slug": "translation", "name": "Translation" } ], "rootPath": true }, { "slug": "monsterinsights", "name": "MonsterInsights", "description": "MonsterInsights is the most popular Google Analytics plugin for WordPress.", "confidence": 100, "version": "8.25.0", "icon": "MonsterInsights.png", "website": "https://www.monsterinsights.com", "cpe": null, "categories": [ { "id": 87, "slug": "wordpress-plugins", "name": "WordPress plugins" }, { "id": 10, "slug": "analytics", "name": "Analytics" } ], "rootPath": true }, { "slug": "scrollreveal", "name": "scrollreveal", "description": null, "confidence": 100, "version": null, "icon": "scrollreveal.svg", "website": "https://scrollrevealjs.org", "cpe": null, "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "lodash", "name": "Lodash", "description": "Lodash is a JavaScript library which provides utility functions for common programming tasks using the functional programming paradigm.", "confidence": 100, "version": null, "icon": "Lodash.svg", "website": "https://www.lodash.com", "cpe": "cpe:2.3:a:lodash:lodash:*:*:*:*:*:*:*:*", "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "cdnjs", "name": "cdnjs", "description": "cdnjs is a free distributed JS library delivery service.", "confidence": 100, "version": null, "icon": "cdnjs.svg", "website": "https://cdnjs.com", "cpe": null, "categories": [ { "id": 31, "slug": "cdn", "name": "CDN" } ], "rootPath": true }, { "slug": "swiper", "name": "Swiper", "description": "Swiper is a JavaScript library that creates modern touch sliders with hardware-accelerated transitions.", "confidence": 100, "version": null, "icon": "Swiper.svg", "website": "https://swiperjs.com", "cpe": null, "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "slider-revolution", "name": "Slider Revolution", "description": "Slider Revolution is a flexible and highly customisable slider.", "confidence": 100, "version": "6.5.12", "icon": "Slider Revolution.svg", "website": "https://www.sliderrevolution.com", "cpe": null, "categories": [ { "id": 5, "slug": "widgets", "name": "Widgets" }, { "id": 7, "slug": "photo-galleries", "name": "Photo galleries" } ], "rootPath": true }, { "slug": "select2", "name": "Select2", "description": "Select2 is a jQuery based replacement for select boxes. It supports searching, remote data sets, and infinite scrolling of results.", "confidence": 100, "version": null, "icon": "Select2.png", "website": "https://select2.org/", "cpe": null, "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "jquery-ui", "name": "jQuery UI", "description": "jQuery UI is a collection of GUI widgets, animated visual effects, and themes implemented with jQuery, Cascading Style Sheets, and HTML.", "confidence": 100, "version": "1.12.1", "icon": "jQuery UI.svg", "website": "https://jqueryui.com", "cpe": "cpe:2.3:a:jquery:jquery_ui:*:*:*:*:*:*:*:*", "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "jquery-migrate", "name": "jQuery Migrate", "description": "Query Migrate is a javascript library that allows you to preserve the compatibility of your jQuery code developed for versions of jQuery older than 1.9.", "confidence": 100, "version": "3.4.1", "icon": "jQuery.svg", "website": "https://github.com/jquery/jquery-migrate", "cpe": null, "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "jquery", "name": "jQuery", "description": "jQuery is a JavaScript library which is a free, open-source software designed to simplify HTML DOM tree traversal and manipulation, as well as event handling, CSS animation, and Ajax.", "confidence": 100, "version": "3.7.1", "icon": "jQuery.svg", "website": "https://jquery.com", "cpe": "cpe:2.3:a:jquery:jquery:*:*:*:*:*:*:*:*", "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "google-analytics", "name": "Google Analytics", "description": "Google Analytics is a free web analytics service that tracks and reports website traffic.", "confidence": 100, "version": "GA4", "icon": "Google Analytics.svg", "website": "https://google.com/analytics", "cpe": null, "categories": [ { "id": 10, "slug": "analytics", "name": "Analytics" } ], "rootPath": true }, { "slug": "core-js", "name": "core-js", "description": "core-js is a modular standard library for JavaScript, with polyfills for cutting-edge ECMAScript features.", "confidence": 100, "version": "3.8.0", "icon": "core-js.png", "website": "https://github.com/zloirock/core-js", "cpe": null, "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "tippy-js", "name": "Tippy.js", "description": "Tippy.js is the complete tooltip, popover, dropdown, and menu solution for the web, powered by Popper.", "confidence": 100, "version": null, "icon": "Tippy.js.svg", "website": "https://atomiks.github.io/tippyjs", "cpe": null, "categories": [ { "id": 59, "slug": "javascript-libraries", "name": "JavaScript libraries" } ], "rootPath": true }, { "slug": "priority-hints", "name": "Priority Hints", "description": "Priority Hints exposes a mechanism for developers to signal a relative priority for browsers to consider when fetching resources.", "confidence": 100, "version": null, "icon": "Priority Hints.svg", "website": "https://wicg.github.io/priority-hints/", "cpe": null, "categories": [ { "id": 92, "slug": "performance", "name": "Performance" } ], "rootPath": true }, { "slug": "font-awesome", "name": "Font Awesome", "description": "Font Awesome is a font and icon toolkit based on CSS and Less.", "confidence": 100, "version": null, "icon": "Font Awesome.svg", "website": "https://fontawesome.com/", "cpe": null, "categories": [ { "id": 17, "slug": "font-scripts", "name": "Font scripts" } ], "rootPath": true }, { "slug": "rss", "name": "RSS", "description": "RSS is a family of web feed formats used to publish frequently updated works—such as blog entries, news headlines, audio, and video—in a standardized format.", "confidence": 100, "version": null, "icon": "RSS.svg", "website": "https://www.rssboard.org/rss-specification", "cpe": null, "categories": [ { "id": 19, "slug": "miscellaneous", "name": "Miscellaneous" } ], "rootPath": true }, { "slug": "open-graph", "name": "Open Graph", "description": "Open Graph is a protocol that is used to integrate any web page into the social graph.", "confidence": 100, "version": null, "icon": "Open Graph.png", "website": "https://ogp.me", "cpe": null, "categories": [ { "id": 19, "slug": "miscellaneous", "name": "Miscellaneous" } ], "rootPath": true } ], "patterns": { "Nginx": [ { "type": "headers", "regex": "nginx(?:\\/([\\d.]{1,250}))?", "value": "nginx", "origKey": "server", "match": "nginx", "confidence": 100, "version": "\\1", "implies": [], "excludes": [] } ], "Cart Functionality": [ { "type": "dom", "regex": "(?:)", "value": "", "origKey": "a[href*='/cart']", "match": "", "confidence": 100, "version": "", "implies": [], "excludes": [] } ], "Font Awesome": [ { "type": "dom", "regex": "(?:([\\d\\.]{1,250})\\/)?(?:css\\/)?font-awesome(?:\\.min)?\\.css", "value": "https://vuuniversitypress.com/wp-content/plugins/woocommerce-ajax-filters/berocket/assets/css/font-awesome.min.css?ver=6.4.3", "origKey": "link[href*='awesome']", "match": "css/font-awesome.min.css", "confidence": 100, "version": "\\1", "implies": [], "excludes": [] } ], "Open Graph": [ { "type": "dom", "regex": "(?:)", "value": "", "origKey": "meta[property*='og:']", "match": "", "confidence": 100, "version": "", "implies": [], "excludes": [] } ], "Priority Hints": [ { "type": "dom", "regex": "(?:)", "value": "", "origKey": "iframe[fetchpriority], img[fetchpriority], script[fetchpriority], link[fetchpriority]", "match": "", "confidence": 100, "version": "", "implies": [], "excludes": [] } ], "RSS": [ { "type": "dom", "regex": "application\\/(?:rss|atom)\\+xml", "value": "application/rss+xml", "origKey": "link[type*='application']", "match": "application/rss+xml", "confidence": 100, "version": "", "implies": [], "excludes": [] } ], "Tippy.js": [ { "type": "dom", "regex": "(?:)", "value": "", "origKey": "style[data-tippy-stylesheet]", "match": "", "confidence": 100, "version": "", "implies": [], "excludes": [] }, { "type": "js", "regex": "(?:)", "value": true, "origKey": "tippy.defaultProps", "match": "", "confidence": 100, "version": "", "implies": [], "excludes": [] } ], "Yoast SEO": [ { "type": "dom", "regex": "(?:)", "value": "yoast-schema-graph", "origKey": "script.yoast-schema-graph", "match": "", "confidence": 100, "version": "", "implies": [ "WordPress" ], "excludes": [] }, { "type": "html", "regex": "