search

richardvenneman / instagrammer

📷 Instagrammer lets you fetch Instagram user info and posts
MIT License
6 stars 5 forks source link

Randomize user agent #5

Open richardvenneman opened 5 years ago

richardvenneman commented 5 years ago

I'm getting the idea that Instagram might not be serving content based on our headless user agent. Investigate if it's possible to randomise the user agent to increase success rate.

mattnwa commented 5 years ago

Hey Richard, Great work! I'm working through this precise problem on our daily collection. It seems like my lambda service was working fine using a mobile user agent until this last weekend. Now 6 of 10 calls are redirecting to the home login page.

Maybe Instagram is catching the server IP? I'm not familiar enough to know that proxies would help in this situation but figure maybe?

richardvenneman commented 5 years ago

Hi @mattnwa thanks for the report!

I suspected the same thing. Instagram could be blacklisting or rate-limiting the IP addresses.

I think a proxy-kind of setup could definitely be a solution to this issue. I've also briefly looked at utilising the Tor network, however I not really familiar with both myself either :)

Will have to investigate the possible options. This gem looks pretty interesting: https://github.com/joshweir/scraypa.