"panic: bad request: 429 Too Many Requests: Throttled"

[novoid]

Hi, Since this project is quite new, I'm not sure about the maturity of the source. However, I've tried it on Xubuntu 18.04 LTS (it fails to install on Debian GNU/Linux 9 Stretch - I may issue a ticket if you want) and I can't generate statistics for my account: https://mastodon.social/@publicvoit

vk@floyd ~/go/bin % ./statootstics
Which instance to connect to (e.g. https://mastodon.social): https://mastodon.social
Loading toots for publicvoit                                                      364 of 4069 [##>--------------------------]   8.95%panic: bad request: 429 Too Many Requests: Throttled

goroutine 1 [running]:
main.parseToot(0xc420168f00, 0xc420669f30, 0x0, 0x0)
        /home/vk/go/src/github.com/muesli/statootstics/stats.go:63 +0x929
main.main()
        /home/vk/go/src/github.com/muesli/statootstics/main.go:153 +0x3b1
2 vk@floyd ~/go/bin %

[muesli]

I'm a bit surprised this happens with mastodon.social for you, as it usually copes with much higher API request loads. Depending on your instance, you might have to adjust the timeout between requests: edit line 173 of main.go and adjust the value of 1000 to something like 2500 (maybe?).

Eventually we should/will add a backoff algorithm around the API requests.

[novoid]

Thanks, muesli, for the hint. Since this is my first encounter with Go: what do I have to do after modifying the source to re-generate the binary? (The only thing I did before was the go get [github] command and this was straight forward.)

I tried go build . in the src directory which generated a new binary. However, it got the same behavior as before so I'm not sure whether or not I re-compiled with the changes.

[muesli]

All you need to do now is run go build. This will compile a new binary.

[muesli]

I've implemented a backoff algorithm in go-mastodon: https://github.com/mattn/go-mastodon/pull/85

Should it get merged, it would fix this issue here.

[novoid]

OK, then the issue persists also with a value of 5000: I get the error always after fetching around 350 toots (~8.8%). I'll try your update.

[novoid]

I'm going to retry for verifying the fix although I've found statootsics which is working fine for my original use-case.

vk@floyd ~/go/src/github.com/muesli % git clone https://github.com/muesli/mastotool.git

Can't compile:

vk@floyd ~/go/src/github.com/muesli/mastotool (git)-[master] % go build
main.go:10:2: cannot find package "github.com/spf13/cobra" in any of:
        /usr/lib/go-1.10/src/github.com/spf13/cobra (from $GOROOT)
        /home/vk/go/src/github.com/spf13/cobra (from $GOPATH)
1 vk@floyd ~/go/src/github.com/muesli/mastotool (git)-[master] %

Resolving dependencies can be solved by using go get instead of git clone.

Removing directory again and testing go get instead:

vk@floyd ~/go/src/github.com/muesli % go get github.com/muesli/mastotool
# github.com/muesli/mastotool
mastotool/search.go:42:5: pg.MinID undefined (type mastodon.Pagination has no field or method MinID)
mastotool/stats.go:86:5: pg.MinID undefined (type mastodon.Pagination has no field or method MinID)
2 vk@floyd ~/go/src/github.com/muesli %

Failed.

Trying to fix with go get -fix:

vk@floyd ~/go/src/github.com/muesli/mastotool (git)-[master] % go get -fix
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
warning: no cgo types: exit status 1
# github.com/muesli/mastotool
./search.go:42:5: pg.MinID undefined (type mastodon.Pagination has no field or method MinID)
./stats.go:86:5: pg.MinID undefined (type mastodon.Pagination has no field or method MinID)
2 vk@floyd ~/go/src/github.com/muesli/mastotool (git)-[master] %

I give up: I just don't have the required knowledge of the go environment to help myself. :-(

Feel free to close this issue if you think that it's resolved. I can't confirm for now.

[muesli]

@novoid Try go get -u -v github.com/muesli/mastotool to also update the dependencies of mastotool.

Alternatively, with Go 1.11+ just clone mastotool anywhere and run

GO111MODULE="on" go build

in the mastotool directory.