scrapfly / scrapfly-scrapers

Web scrapers for popular targets powered Scrapfly.io
https://scrapfly.io
Other
166 stars 46 forks source link

running the glassdoor scraper #2

Closed leo307646595 closed 10 months ago

leo307646595 commented 10 months ago

Hi,

I am trying to use glassdoor scraper to scrape companies review information. However, when I try to run the asyncio.run it alway comes with "asyncio.run() cannot be called from a running event loop".

I am wondering if you could help me?

many thanks

Granitosaurus commented 10 months ago

How are you running your Python script? Some environtments like notebooks etc. already have asyncio loop running so you can simply await run() instead or retrieve the loop using:

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

I'd recommend getting familiar with Python's asyncio as it's a vital tool for web scraping and there a lot of great tutorials out there! I wrote a short introduction here: https://scrapfly.io/blog/web-scraping-speed/#async-requests