response status 429 Too Many Requests: Rate limit exceeded

[Volvalzark]

この問題を解決するために、APIリクエストに秒単位の遅延を設定することは可能ですか？

[mmpx12]

@Volvalzark english plz

[Volvalzark]

Sorry, I thought I sent it in English, but I sent it in Japanese. Is it possible to set a delay in seconds for API requests to solve this problem?

[Kameshennyn]

Same issue there, only happens once logged

[mmpx12]

Since i'm using go routine, i first need to set the number of go routines it used before setting up a delay (without adding delay will change nothing).

[Volvalzark]

I don't know the appropriate number of go routines, so I can't really say, how many is appropriate?

[gzygood]

“response status 429 Too Many Requests: Rate limit exceeded” Yes, it's easy to see this!

[shunnag]

I am modifying it as follows.

    wg := sync.WaitGroup{}
    for tweet := range scraper.GetTweets(context.Background(), usr, nbrs) {
        time.Sleep(2 * time.Second) // Sleep after each tweet is fetched 
        if tweet.Error != nil {
            wg.Wait() //Wait to prevent program from exiting during image acquisition
            fmt.Println(tweet.Error)
            os.Exit(1)
        }
        if vidz {
            time.Sleep(5 * time.Second) // Extra long sleep when acquiring video 
            wg.Add(1)
            go videoUser(&wg, tweet, output, retweet)
        }
        if imgs {
            time.Sleep(1 * time.Second) //A little extra sleep during image acquisition
            wg.Add(1)
            go photoUser(&wg, tweet, output, retweet)
        }
    }
    wg.Wait()

[mmpx12]

@shunnag you can also use: scraper.WithDelay(2) instead of time.Sleep(2 * time.Second) // Sleep after each tweet is fetched.

but for:

        if tweet.Error != nil {
+           wg.Wait() //Wait to prevent program from exiting during image acquisition
            fmt.Println(tweet.Error)
            os.Exit(1)
        }
        if vidz {
+           time.Sleep(5 * time.Second) // Extra long sleep when acquiring video 
            wg.Add(1)
            go videoUser(&wg, tweet, output, retweet)
        }
        if imgs {
+           time.Sleep(1 * time.Second) //A little extra sleep during image acquisition
            wg.Add(1)
            go photoUser(&wg, tweet, output, retweet)
        }
    }

did you tried ans Is it working?

[shunnag]

@mmpx12 It appears to be functioning correctly. There is a 2-second delay applied to all API accesses, and an additional delay is added when fetching tweets with the "imgs, vids" option enabled, even if the tweet itself doesn't contain any images or videos. We could potentially adjust the numbers to align with the 429 rate limit in order to increase the acquisition speed, but for now, it's running smoothly without causing errors, albeit at a slower pace.

[mmpx12]

@shunnag This should be added with a flag like '-S|--slow'. I use a different proxy every time I run twmd (not for a single tweet, but for -a, -v, or -i) so I don't encounter 429 error. Another option should be to add '-f|--fast' to enable goroutine. But I'm not sure if without goroutine and without sleep, 429 error would occur.

[shunnag]

@mmpx12 like this?

-   var nbr, single, output string
+   var nbr, single, output, delay string
    var retweet, all, printversion, nologo, login, twofa bool

    op.On("-B", "--no-banner", "Don't print banner", &nologo)
+   op.On("-d", "--delay SECOND(S)", "Use specified seconds delay for API access", &delay)
    op.Exemple("twmd -u Spraytrains -o ~/Downlaods -a -r -n 300")

    wg := sync.WaitGroup{}

+   if delay == "" {
+       delay = "0"
+   }
+   delay64, _ := strconv.ParseInt(delay, 10, 64)
+
    for tweet := range scraper.GetTweets(context.Background(), usr, nbrs) {

and use time.Sleep(time.Second * time.Duration(delay64)) in loop or scraper.WithDelay(delay64)

[tazeps]

I am also encountering this issue and was thinking of having a delay function. which version is working best for you guys ? I'd wait for a new release if @mmpx12 will incorporate it otherwise I may try to add it. though i'm new to using github for software so i might struggle