mishushakov / llm-scraper

Turn any webpage into structured data using LLMs
MIT License
2.18k stars 140 forks source link

Interacting with the website using playwright #37

Open daonsh opened 1 month ago

daonsh commented 1 month ago

Since playwright is a powerful library and can interact with any element in the page, is it possible to use this library to perform tasks such as:

  1. Open website, click on search form, fill a text, click on search button, scrape the resulting list that is loaded on new page.
  2. Find a button that does X and click on it?

The idea is to scrape job posts from a website, so for that I need to ask which button will lead to the jobs/careers page, click on it, open new page, filter jobs to a specific country, click on search button, scrape resulting list.

mishushakov commented 4 weeks ago

PRs are welcome 😉