anilabhadatta / educative.io_scraper

Educative.io Course Downloader developed using Python and Selenium. Refer Readme.md for setup instructions.
MIT License
163 stars 55 forks source link

incompatible requirements #135

Open abhinavm24 opened 1 day ago

abhinavm24 commented 1 day ago

The conflict is caused by: The user requested urllib3==2.2.1 requests 2.31.0 depends on urllib3<3 and >=1.21.1 seleniumbase 4.24.11 depends on urllib3<2 and >=1.26.18; python_version < "3.10"

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip to attempt to solve the dependency conflict

Facing this on mac M1, latest available python-tk

abhinavm24 commented 1 day ago

nvm, worked with clean install

abhinavm24 commented 1 day ago

dumping all the things that I found useful while scrapping this time

To scrape all course list- scroll through the course list on https://www.educative.io/search?tab=courses and execute on console `const courseUrls = Array.from(document.querySelectorAll('.rounded a')) .map(a => a.href) .filter(url => url.includes('/courses/') || url.includes('/path/') || url.includes('/assessments/'));

const csvContent = "data:text/csv;charset=utf-8," + courseUrls.join("\n"); const encodedUri = encodeURI(csvContent); const link = document.createElement("a"); link.setAttribute("href", encodedUri); link.setAttribute("download", "course_urls.csv"); document.body.appendChild(link); link.click();`

abhinavm24 commented 1 day ago

first page of most courses is course url/about-the-course

abhinavm24 commented 1 day ago

nvm, scrape all course url does the same, but slow

abhinavm24 commented 1 day ago

is it possible to run multiple instances of scapper on same machine?