search

philippta / flyscrape

Flyscrape is a command-line web scraping tool designed for those without advanced programming skills.
https://flyscrape.com
Mozilla Public License 2.0
1.04k stars 31 forks source link

How to render Javascript websites in this scraper #28

Closed labi1240 closed 8 months ago

labi1240 commented 9 months ago

I an trying to scrape the website which required the page to render in browser but facing this issue failed to compile script: running user script: GoError: Invalid module at github.com/dop251/goja_nodejs/require.(*RequireModule).require-fm (native)

I am using selenium grid with docker and trying to load the page

can you help

philippta commented 9 months ago

I am assuming you are trying to import a selenium grid package into your scraping script, which is not supported. You can only import the "flyscrape" packages.

However to support JavaScript rendering I am currently working on a hosted chrome renderer that can be used with flyscrape. It is currently still very experimental but I’m always looking for testers and collect some feedback.

You can try it out by adding this to your script config:

proxies: ["http://shared-user-until-feb-2024@eu.proxy.flyscrape.com:3128/"],

Please let me know if that works for you.

philippta commented 8 months ago

Closing because of inactivity.