muhamed-didovic / ccdown

Scraper and downloader for codecourse.com
MIT License
8 stars 2 forks source link

ERROR 1015: limit reached #9

Closed nickstarkloff closed 10 months ago

nickstarkloff commented 1 year ago

Hello, after a short time I always get this error. The script crashes after that.

MAIN errorr in cli.js: Error: ERROR 1015: limit reached
muhamed-didovic commented 1 year ago

hey @nickstarkloff, it seems we need to slow down on download, there are limits on how many files can be downloaded, I'll see to slow down the downloader...

connecteev commented 1 year ago

@muhamed-didovic Please make this an optional flag for those where this is not a problem. I, for one, think the downloads take a long time to finish already.

connecteev commented 1 year ago

Also, @nickstarkloff if you do see this, just wait for some time and try again....you may be hitting a rate limit. I would suggest waiting an hour to see if that fixes it.

muhamed-didovic commented 1 year ago

@connecteev you can always make new downloader yourself :)

connecteev commented 1 year ago

@muhamed-didovic I don't have the talent to create a downloader like you. Anyway it was just a suggestion.

muhamed-didovic commented 1 year ago

@nickstarkloff pull or update to new version, download is slower now but you should not get 1015 error @connecteev I added also source code download for courses as well

connecteev commented 1 year ago

@muhamed-didovic thanks for the updates. It would be great to have an option to turn the throttling off so that ccdown doesn't slow down. Possible to add that as a command-line option?

connecteev commented 1 year ago

Btw, with the new version of ccdown

$ccdown --version
0.20.2

I get this error: image

ccdown ✔ Do you want all courses? … no ✔ Choose "Y" if you want to search for a course otherwise choose "N" if you have a link for download … no ✔ Enter url for download. … https://codecourse.com/courses/eloquent-relationships-by-example ✔ Enter email … testemail@gmail.com ✔ Enter password … *** ✔ Enter a directory to save a file (eg: /Users/kp/Code/web/_course_downloaders) … /Users/kp/Code/web/_course_downloaders/videos ✔ Login... completed. ✓ Found: 1 courses ⠋ Extracting 1/1 course: eloquent-relationships-by-example has 1/33 episodes, scraping url: https://codecourse.com/watch/eloquent-relationships-by-example?part=installation- database-and-migration-refresher ⠋ to be processed by yt-dlp... 00-installation-database-and-migration-refresher.mp4

Seems to be something security related and this is the same error referenced here https://discussions.apple.com/thread/253741610

This prompt prevents the downloads from continuing, and btw does not accept my Mac's system / root password. Clicking allow or always allow" requires a password, which does not accept my Mac's system / root password. Clicking deny just makes the script wait forever and do nothing.

connecteev commented 1 year ago

I seem to have fixed the "keychain" error (which seems to be triggered every time I run ccdown) by following the instructions here: https://discussions.apple.com/thread/253741610?answerId=253741610021#253741610021

However, the script still doesnt download any videos (or, it's taking a REALLY long time to get started). Would be great to have an option to turn off throttling / this new feature @muhamed-didovic. Many thanks..

update: eventually the script failed with this error:

✔ Do you want all courses? … no
✔ Choose "Y" if you want to search for a course otherwise choose "N" if you have a link for download … no
✔ Enter url for download. … https://codecourse.com/courses/eloquent-relationships-by-example
✔ Enter email … testemail@gmail.com
✔ Enter password … ***************
✔ Enter a directory to save a file (eg: /Users/kp/Code/web/_course_downloaders) … /Users/kp/Code/web/_course_downloaders/videos
✔ Login...  completed.
✓ Found: 1 courses
MAIN errorr in cli.js: Error: A spinner named '/Users/kp/Code/web/_course_downloaders/videos/517-Eloquent Relationships By Example/00-installation-database-and-migration-refresher.mp4' already existse-and-migration-refresher
    at Spinnies.add (/opt/homebrew/lib/node_modules/ccdown/node_modules/dreidels/dist/index.js:409:19)
    at default (file:///opt/homebrew/lib/node_modules/ccdown/lib/helpers/downOverYoutubeDL.js:63:8)
    at Promise.map.concurrency (file:///opt/homebrew/lib/node_modules/ccdown/lib/Crawler.js:942:19)
From previous event:
    at process.processImmediate (node:internal/timers:478:21)
From previous event:
    at Crawler.d (file:///opt/homebrew/lib/node_modules/ccdown/lib/Crawler.js:929:30)
    at Promise.all.Promise
nickstarkloff commented 1 year ago

Thanks for the update!

It ran fine for a while, but then unfortunately the error started again.

Maybe catch this error, wait for some time and then run crawling again? Otherwise we have to start the script again and again if this error appears.

muhamed-didovic commented 1 year ago

@nickstarkloff it is possible that I add a 'slow' downloader, but the download will be quite slow then? @connecteev added back the concurrency flag now you can control how many files or videos you want to download concurrently

connecteev commented 1 year ago

Thanks @muhamed-didovic!