scrapfly / scrapfly-scrapers

Scalable Python web scraping scripts for +40 popular domains
https://scrapfly.io
Other
312 stars 88 forks source link

running the glassdoor scraper #2

Closed leo307646595 closed 1 year ago

leo307646595 commented 1 year ago

Hi,

I am trying to use glassdoor scraper to scrape companies review information. However, when I try to run the asyncio.run it alway comes with "asyncio.run() cannot be called from a running event loop".

I am wondering if you could help me?

many thanks

Granitosaurus commented 1 year ago

How are you running your Python script? Some environtments like notebooks etc. already have asyncio loop running so you can simply await run() instead or retrieve the loop using:

loop = asyncio.get_event_loop()
loop.run_until_complete(run())

I'd recommend getting familiar with Python's asyncio as it's a vital tool for web scraping and there a lot of great tutorials out there! I wrote a short introduction here: https://scrapfly.io/blog/web-scraping-speed/#async-requests