rojter-tech / pluradl.py

Automated download of Pluralsight courses
MIT License
2 stars 0 forks source link

HTTPError 49 too many requests #26

Open spoiledtechie opened 4 years ago

spoiledtechie commented 4 years ago

Ive let Pluradl run over the weekend until today. Its done quite well at doing what was promised. THANK YOU.

But I did hit the 429 error today and have been suspended from my account. I was using the default settings and rate limiting as explained in the readme. But it said to report it here so that is what I am doing.

I want to contact and ask them to unsuspend. But I need to increase the limiting most likely, right?

prikhi commented 4 years ago

Were you downloading 24/7 for multiple days? I bumped the time limits & have been limiting my downloading to ~6hrs/day to try and avoid a ban.

geffchang commented 4 years ago

@prikhi Is it safe to download 1 course per day everyday? Also, are you using cookies, or are you using the username/password (i.e. the default command described in the README)?

prikhi commented 4 years ago

I do 1 or 2 courses a day, but only been using this for ~3 days now. I haven't tried using cookies. Can I just grab them out of dev tools? I assumed that the script would store the cookies itself, but never verified...

I also applied these changes to slow everything down because I've only got a handful of courses I want to grab:

diff --git a/plura_dl/extractor/pluralsight.py b/plura_dl/extractor/pluralsight.py
index c780b24..1adaa3e 100644
--- a/plura_dl/extractor/pluralsight.py
+++ b/plura_dl/extractor/pluralsight.py
@@ -393,7 +393,7 @@ query viewClip {
                 # To somewhat reduce the probability of these consequences
                 # we will sleep random amount of time before each call to ViewClip.
                 self._sleep(
-                    random.randint(2, 5), display_id,
+                    random.randint(2, 10), display_id,
                     '%(video_id)s: Waiting for %(timeout)s seconds to avoid throttling')

                 if not viewclip:
diff --git a/pluradl.py b/pluradl.py
index 7f679b0..abb9202 100644
--- a/pluradl.py
+++ b/pluradl.py
@@ -9,10 +9,10 @@ certpath = os.path.abspath(certifi.where())
 os.environ["SSL_CERT_FILE"] = certpath

 # IMPORTANT SETTINGS TO PREVENT SPAM BLOCKING OF YOUR ACCOUNT/IP AT PLURALSIGHT # # # # # # # # # #
-SLEEP_INTERVAL = 60     # minimum sleep time                   #                                  #
-SLEEP_OFFSET   = 120    # set random sleep time up to          #  Change this at your own risk.   #
-SLEEP_PLAYLIST = 60     # sleep time between playlist requests #                                  #
-RATE_LIMIT     = 10**6  # download rate in bytes/s             #                                  #
+SLEEP_INTERVAL = 90     # minimum sleep time                   #                                  #
+SLEEP_OFFSET   = 240    # adding random sleep time up to       #  Change this at your own risk.   #
+SLEEP_PLAYLIST = 120    # sleep time between playlist requests #                                  #
+RATE_LIMIT     = (0.6 * 10**6)  # download rate in bytes/s     #                                  #
 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

 # Global defaults
geffchang commented 4 years ago

@prikhi I have no idea if the script gets cookies; I haven't confirmed too. If you are using Chrome, you can get the cookies using this addon: https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg?hl=en

CarlosJoseChaconChavarria commented 4 years ago

download courses as you would when normally watching from 8 am to 7 or 8 pm you are not machine to watch 24/7 videos so just you have to do it that way

Nocxy commented 4 years ago

download courses as you would when normally watching from 8 am to 7 or 8 pm you are not machine to watch 24/7 videos so just you have to do it that way

How long did you do that? Weren't you banned at least once ?

CarlosJoseChaconChavarria commented 4 years ago

download courses as you would when normally watching from 8 am to 7 or 8 pm you are not machine to watch 24/7 videos so just you have to do it that way

How long did you do that? Weren't you banned at least once ?

not at all I've downloaded like 35 courses so far in like 10 days

Bigemul commented 4 years ago

I downloaded around 80 courses. What are your parameters to avoid being banned ?

CarlosJoseChaconChavarria commented 4 years ago

I downloaded around 80 courses. What are your parameters to avoid being banned ?

well yours are working why do you ask? I use a software name NetLimiter Pro and I see the download speed to 150kbps so the dowload times plus the sleep time is like wathing the course so I download like 8 to 12 hours video a day