raviqqe / muffet

Fast website link checker in Go
MIT License
2.47k stars 95 forks source link

"too many redirections" with Twitter links #308

Closed philrz closed 1 year ago

philrz commented 1 year ago

I've got pages with links to Twitter that have always been checked successfully with Muffet in the past. But starting about a week ago they fail consistently like this:

$ muffet --version
2.8.0

$ muffet --verbose https://twitter.com/twitter
failed to fetch root page: too many redirections

Suffice to say, the links work fine in a browser. I assume something changed on the Twitter side, maybe specifically to foil link checkers. However, before I go excluding all Twitter links from my automation I'm hoping someone who knows the Muffet code could give it a closer look and confirm if there's a way to make it succeed. FWIW, I've already tried some of my favorite workarounds I've used in similar situations in the past, such as setting headers to make my client look more "real":

$ muffet --verbose --header="User-Agent:Mozilla/5.0" --header="Accept-Encoding:zstd,br,gzip,deflate" https://twitter.com/twitter 
failed to fetch root page: too many redirections
raviqqe commented 1 year ago

Thanks for the report! It's fixed on the latest release.

philrz commented 1 year ago

Thanks so much @raviqqe!