I've got pages with links to Twitter that have always been checked successfully with Muffet in the past. But starting about a week ago they fail consistently like this:
$ muffet --version
2.8.0
$ muffet --verbose https://twitter.com/twitter
failed to fetch root page: too many redirections
Suffice to say, the links work fine in a browser. I assume something changed on the Twitter side, maybe specifically to foil link checkers. However, before I go excluding all Twitter links from my automation I'm hoping someone who knows the Muffet code could give it a closer look and confirm if there's a way to make it succeed. FWIW, I've already tried some of my favorite workarounds I've used in similar situations in the past, such as setting headers to make my client look more "real":
$ muffet --verbose --header="User-Agent:Mozilla/5.0" --header="Accept-Encoding:zstd,br,gzip,deflate" https://twitter.com/twitter
failed to fetch root page: too many redirections
I've got pages with links to Twitter that have always been checked successfully with Muffet in the past. But starting about a week ago they fail consistently like this:
Suffice to say, the links work fine in a browser. I assume something changed on the Twitter side, maybe specifically to foil link checkers. However, before I go excluding all Twitter links from my automation I'm hoping someone who knows the Muffet code could give it a closer look and confirm if there's a way to make it succeed. FWIW, I've already tried some of my favorite workarounds I've used in similar situations in the past, such as setting headers to make my client look more "real":