search

zrashwani / arachnid

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites
MIT License
253 stars 60 forks source link

Does not support js-rendered sites #30

Closed mkantautas closed 5 years ago

mkantautas commented 6 years ago

Sites like https://taxibambino.com or other sites that html is being rendered from js is not supported by this crawler and I understand that by design(being a back-end based crawler) this is not fixable. I am afraid that with the lacking support of js-only sites this crawler becomes obsolete.

zrashwani commented 5 years ago

Hello, support of Javascript based sites is now enabled by using enableHeadlessBrowserMode method which use Chrome engine in the background to render Javascript

@neorganic can you check this method and see if it solves your issue?

zrashwani commented 5 years ago

closing for now, since Js-rendered site support is added in release 2.0, feel free to open new issue if there is any other issue