postaddictme / instagram-php-scraper

Get account information, photos, videos, stories and comments.
https://packagist.org/packages/raiym/instagram-php-scraper
MIT License
3.09k stars 800 forks source link

ACCOUNT_JSON_INFO = 'https://www.instagram.com/{username}/?__a=1&__d=dis' no longer working #1134

Open andre-santos1 opened 8 months ago

andre-santos1 commented 8 months ago

Hi there! It looks like that ACCOUNT_JSON_INFO = 'https://www.instagram.com/{username}/?__a=1&__d=dis' is not working anymore.

Response is: Sorry, this page isn't available. The link you followed may be broken, or the page may have been removed. Go back to Instagram.

Does anyone know what I should add to the URL 'https://www.instagram.com/{username}' to get account json info?

Tks!

beshoo commented 8 months ago

I believe we're no longer in a golden age. We're now in a "shit age" due to Instagram's new rules. LOL. This JSON API will end soon!

bjmeo8 commented 8 months ago

Hi there! It looks like that ACCOUNT_JSON_INFO = 'https://www.instagram.com/{username}/?a=1&d=dis' is not working anymore.

Response is: Sorry, this page isn't available. The link you followed may be broken, or the page may have been removed. Go back to Instagram.

Does anyone know what I should add to the URL 'https://www.instagram.com/{username}' to get account json info?

Tks!

Hello, I also have the same issue. I think they have probably changed the magic URL parameters "?a=1&d=dis".

I searched all over the internet and eventually found a solution on Reddit: (https://www.reddit.com/r/webscraping/comments/19fgptv/comment/kjjy2v7/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)

and on this web scraping service's blog: (https://scrapfly.io/blog/how-to-scrape-instagram/)[https://scrapfly.io/blog/how-to-scrape-instagram/)

and their github (https://github.com/scrapfly/scrapfly-scrapers/blob/main/instagram-scraper/instagram.py](https://scrapfly.io/blog/how-to-scrape-instagram/)

To retrieve account information, you should proceed as follows:

curl -X GET "https://i.instagram.com/api/v1/users/web_profile_info/?username=google" \ -H "User-Agent: iphone_ua" \ -H "x-ig-app-id: 936619743392459"

You can try the curl request directly here https://reqbin.com/curl and you will see that the response is a JSON containing the information accessible via the old method "?a=1&d=dis"

beshoo commented 8 months ago

image

bjmeo8 commented 8 months ago

It seems there is a formatting issue with the initial request I submitted. Try this:

curl -X GET "https://i.instagram.com/api/v1/users/web_profile_info/?username=google" \ -H "User-Agent: iphone_ua" \ -H "x-ig-app-id: 936619743392459"

I just tried again, and it's working fine. Here's a screenshot:

Screenshot 2024-01-28 at 11-16-25 Run Curl Commands Online

beshoo commented 8 months ago

It will end soon.

On Sun, Jan 28, 2024 at 1:20 PM bjmeo8 @.***> wrote:

It seems there is a formatting issue with the initial request I submitted. Try this:

curl -X GET " https://i.instagram.com/api/v1/users/web_profile_info/?username=google" -H "User-Agent: iphone_ua" -H "x-ig-app-id: 936619743392459"

I just tried again, and it's working fine. Here's a screenshot:

Screenshot.2024-01-28.at.11-16-25.Run.Curl.Commands.Online.png (view on web) https://github.com/postaddictme/instagram-php-scraper/assets/68184289/5dd81ba7-fe04-4aa9-8f49-cb7593725173

— Reply to this email directly, view it on GitHub https://github.com/postaddictme/instagram-php-scraper/issues/1134#issuecomment-1913545664, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLT2Z5SCOAXULQAIP7MELYQYQ77AVCNFSM6AAAAABCKLT3PKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTGU2DKNRWGQ . You are receiving this because you commented.Message ID: @.***>

bjmeo8 commented 8 months ago

You think? Why?

Perhaps the recent restriction of their public API is related to this?: https://www.mcafee.com/blogs/internet-security/26-billion-records-released-the-mother-of-all-breaches/

Alternatively, they may be concerned that Instagram data is being massively scraped by competitors to train their AI models. Perhaps...

beshoo commented 8 months ago

I have been using an Instagram scraper since 2015, which enabled me to earn $10,000 each month. However, Instagram has recently started using AI to crack down on such activities. Although am were able to bypass the restrictions using residential proxies, Instagram is now taking further measures to put an end to this practice. They started with hashtags and are now targeting users. It seems that even the workarounds will soon be eliminated.

bjmeo8 commented 8 months ago

Okay, I understand your point of view. But does that mean that all the bots, scraping services, etc., in the market will soon go extinct too? Wow, that will be a tough blow for many.

beshoo commented 8 months ago

I agree with you. And yes

On Sun, Jan 28, 2024 at 1:53 PM bjmeo8 @.***> wrote:

Okay, I understand your point of view. But does that mean that all the bots, scraping services, etc., in the market will soon go extinct too? Wow, that will be a tough blow for many.

— Reply to this email directly, view it on GitHub https://github.com/postaddictme/instagram-php-scraper/issues/1134#issuecomment-1913553540, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLT2YYIHGYKC6ITPADJZDYQYU4BAVCNFSM6AAAAABCKLT3PKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTGU2TGNJUGA . You are receiving this because you commented.Message ID: @.***>

kvrushifa commented 8 months ago

@beshoo how were you able to earn that much money just by scraping instagram ? :o

beshoo commented 8 months ago

"I didn't just scrape Instagram, I actually created a website called instaranker.com. It's a service where you can add your Instagram account and it will automatically engage with other users by liking, following, and saving their posts.

I was able to make around $300 to $400 a day from 2016 to 2020. However, things started to change after 2021 website closed at the end of 2021..

I have another website called viralcaption.com that only scrapes data. I had to remove two functions which were user information and tag information. but am sure I will close the service very soon.

andre-santos1 commented 8 months ago

I wasn't able to reproduce your steps. But it worked as an insight for me and I did as follows:

  1. Changed my browser user-agent (try this browser extension for example: https://chromewebstore.google.com/detail/user-agent-switcher-for-c/djflhoibgkdhkhhcedjiklpkjnoahfmg) to this:

Mozilla/5.0 (Linux; Android 9; SM-A102U Build/PPR1.180610.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/74.0.3729.136 Mobile Safari/537.36 Instagram 155.0.0.37.107 Android (28/9; 320dpi; 720x1468; samsung; SM-A102U; a10e; exynos7885; en_US; 239490550)

  1. I pasted (without quotes) "https://i.instagram.com/api/v1/users/web_profile_info/?username={username}" on Chrome and I was able to get the JSON response. ({username} must be replaced by your target username)

  2. To better visualize the results I recommend installing JSON Beautifier & Editor extension on Chrome (https://chromewebstore.google.com/detail/json-beautifier-editor/lpopeocbeepakdnipejhlpcmifheolpl)

jfrenl commented 8 months ago

Yeah i found something called ?user=0 it's probably useless information but maybe someone else can use this. Also ?d=1 has a same kind of response.

beshoo commented 8 months ago

__d has to be dis only. ps: __user=0 did nothing!

jfrenl commented 8 months ago

A bit ago have i always used d=1 to get data too until it got patched and then i moved to d=dis. I am not sure why it has been patched.

beshoo commented 8 months ago

I have been using an Instagram scraper since 2015, which enabled me to earn $10,000 each month. However, Instagram has recently started using AI to crack down on such activities. Although am were able to bypass the restrictions using residential proxies, Instagram is now taking further measures to put an end to this practice. They started with hashtags and are now targeting users. It seems that even the workarounds will soon be eliminated.

@jfrenl

jfrenl commented 8 months ago

Yeah i read it, but still unfortunate that this is happening. I've been looking for solutions for a 2 weeks now. I hope to have a break trough soon.

jfrenl commented 8 months ago

I heard you were talking about proxies and i was wondering if you had any cheap providers for proxies. Because it doesn't matter if it is residential proxies or datacenter ips.

jfrenl commented 7 months ago

@beshoo I am not sure if you do know something about this but somehow this instagram url https://www.instagram.com/nike/?__a=1&__d=dis response back as status code 201. But normally you would get a 404 page from instagram.

beshoo commented 7 months ago

Even though, there is no json response....

On Wed, Feb 14, 2024, 8:15 PM jfrenl @.***> wrote:

@beshoo https://github.com/beshoo I am not sure if you do know something about this but somehow this instagram url https://www.instagram.com/nike/?__a=1&__d=dis response back as status code 201. But normally you would get a 404 page from instagram.

— Reply to this email directly, view it on GitHub https://github.com/postaddictme/instagram-php-scraper/issues/1134#issuecomment-1944265062, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLT225ORLZ45MJ3WYNRQ3YTTWMNAVCNFSM6AAAAABCKLT3PKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBUGI3DKMBWGI . You are receiving this because you were mentioned.Message ID: @.***>

jfrenl commented 7 months ago

Yeah that's right but any idea why statuscode 201 created?

beshoo commented 7 months ago

I don't know, but as I said, its helpless...

On Wed, Feb 14, 2024, 8:23 PM jfrenl @.***> wrote:

Yeah that's right but any idea why statuscode 201 created?

— Reply to this email directly, view it on GitHub https://github.com/postaddictme/instagram-php-scraper/issues/1134#issuecomment-1944276733, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLT2ZASB53BETR4OZO3YDYTTXIDAVCNFSM6AAAAABCKLT3PKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBUGI3TMNZTGM . You are receiving this because you were mentioned.Message ID: @.***>

megalancast commented 3 months ago

I am wondering how paid APIs like https://rapidapi.com/neotank/api/instagram130 work? I've checked it for a week and it seems to work pretty good, and /account-info endpoint returns exactly what https://i.instagram.com/api/v1/users/web_profile_info/ endpoint returns (I think)...

jfrenl commented 3 months ago

I am wondering how paid APIs like https://rapidapi.com/neotank/api/instagram130 work? I've checked it for a week and it seems to work pretty good, and /account-info endpoint returns exactly what https://i.instagram.com/api/v1/users/web_profile_info/ endpoint returns (I think)...

I am pretty sure the person uses lots of accounts in the backend and due to people paying every month he can keep this project up. But I have a different solution that doesn't cost me anything.