postaddictme / instagram-php-scraper

Get account information, photos, videos, stories and comments.
https://packagist.org/packages/raiym/instagram-php-scraper
MIT License
3.1k stars 800 forks source link

Get 404 when use function getMediasByTag #70 #71

Closed senol01 closed 7 years ago

senol01 commented 7 years ago

InstagramScraper\Exception\InstagramException Object ( [message:protected] => Response code is 404. Body:

Unbelievable, but the problem is solved if you install cookies (I loaded https://www.instagram.com/explore/tags/video/?__a=1&max_id=1 and copied them from the query, then inserted in /Unirest/Request.php on line 236)

$headers = [ 'cookie' => 'simple cookies' ];

public static function get($url, $headers = array(), $parameters = null, $username = null, $password = null) { $headers = [ 'cookie' => 'simple cookies' ]; return self::send(Method::GET, $url, $parameters, $headers, $username, $password); }

`require '/vendor/autoload.php';

use InstagramScraper\Instagram;

$medias = Instagram::getMediasByTag('zara', 30); echo json_encode($medias);`

raiym commented 7 years ago

Hello @senol01 thank you.

It seems that Instagram started require auth for those two links:

    const MEDIA_JSON_BY_LOCATION_ID = 'https://www.instagram.com/explore/locations/{{facebookLocationId}}/?__a=1&max_id={{maxId}}'; // doesn't work
    const MEDIA_JSON_BY_TAG = 'https://www.instagram.com/explore/tags/{tag}/?__a=1&max_id={max_id}'; // doesn't work
filippodicostanzo commented 7 years ago

From today instagram disabled access at your website without login.

There is any solution for bypass this problem?

filippodicostanzo commented 7 years ago

@senol01 how to install the cookie?

I insert the $headers = [ 'cookie' => 'simple cookies' ]; in the specific line but i recive the same error!

InstagramScraper\Exception\InstagramException Object ( [message:protected] => Response code is 404. Body:

Any solution for this issue?

pinotarantino commented 7 years ago

This work : https://www.instagram.com/explore/tags/fitness/?__a=1&max_id=1

Tried at 17.30

filippodicostanzo commented 7 years ago

@pinotarantino how can i try?

where I insert the url?

Do you can specific me all the steps?

If you're aren't logged in instagram the url isn't working. And when i call the function in my php file return InstagramScraper\Exception\InstagramException Object ( [message:protected] => Response code is 404. Body:

pinotarantino commented 7 years ago

if you click on the link I posted above it works, also https://www.instagram.com/explore/tags/fitness/?__a=1 works; you get the json data... if this works, maybe they're stopping instagram-php-scraper? I don't know...

filippodicostanzo commented 7 years ago

If you are not logged into instagram the link doesn't work and return "Spiacenti, questa pagina non è disponibile È possibile che il link che hai seguito sia corrotto o che la pagina sia stata rimossa. Torna su Instagram."

maxmaxme commented 7 years ago

@fabianoroberto

  1. Open https://www.instagram.com/explore/tags/fitness/?__a=1&max_id=1
  2. Copy all cookies
  3. Insert in /Unirest/Request.php on line 236 this code: $headers = [ 'cookie' => 'simple cookies' ];

Sample: 2017-03-15 19 44 13

filippodicostanzo commented 7 years ago

@maxmaxme how can i get this string for the cookie?

maxmaxme commented 7 years ago

@filippodicostanzo

  1. F12 (CMD+Option+I)
  2. Console
  3. Write: console.log(document.cookie)
filippodicostanzo commented 7 years ago

@maxmaxme thank you but for me doesn't work

return anytime 404 :(

senol01 commented 7 years ago

maxmaxme did you try if it works? console.log(document.cookie) mid=WMl5gAAEAAHXmW0Bp5VYmEgeERhA; ig_pr=1; ig_vw=1366; ds_user_id=4211931173; s_network=""; csrftoken=BREKQUyRvTIOfomdbj2WvpYlAhZippmw have only become this

maxmaxme commented 7 years ago

@filippodicostanzo @senol01 Sorry. I took cookies from another place.

  1. Open https://www.instagram.com/explore/tags/fitness/?__a=1&max_id=1
  2. F12 (CMD+Option+I)
  3. Network
  4. F5
  5. Click on "?__a=1&max_id=1"
  6. Copy Cookies from "Request Headers"

2017-03-15 20 39 48

senol01 commented 7 years ago

Bro thank you very much https://www.instagram.com/explore/tags/usa/?__a=1 But i have one questition more, how i can find the popular Medias?

maxmaxme commented 7 years ago

@senol01 Ask this question to the developer @postaddictme

senol01 commented 7 years ago

for maxmaxme

pinotarantino commented 7 years ago

Thanks to senol01 and maxmaxme, your solution worked for me...! ;) Do you think that this escamotage could resist for long time to Instagram countermeasure? Anyway, Thank you very much!

filippodicostanzo commented 7 years ago

@maxmaxme & @senol01 thank you're saved my week :)

Anynone knows the duration of these cookie?

ximrx commented 7 years ago

This is not a solution, is this library going to work by adding authentication and API key or is it going to be shut down?

raiym commented 7 years ago

Thanks for support guys. Really appreciate it. For now only to urls are require auth:

const MEDIA_JSON_BY_LOCATION_ID = 'https://www.instagram.com/explore/locations/{{facebookLocationId}}/?__a=1&max_id={{maxId}}'; // doesn't work
const MEDIA_JSON_BY_TAG = 'https://www.instagram.com/explore/tags/{tag}/?__a=1&max_id={max_id}'; // doesn't work

related to tags, locations and get account by id.

All other functions work as before. (at least for now) Of course I do understand needs about tags and locations.

We already have private instagram api: https://github.com/mgp25/Instagram-API developed by @mgp25

So I don't know for now is postaddict library needs auth or not?

Actually, I will be glad to hear your suggestions and feeback

raiym commented 7 years ago

Well. I have decided that library will support web login and scraping tags and locations will be fixed

And I think about adding official API support as well

filippodicostanzo commented 7 years ago

@raiym thank you! after this intensive work you can explain how to install this for scraping tags please :)

aik27 commented 7 years ago

Seems, this query (for tagged data) work without any authorization (only csrftoken cookie required).

ig_hashtag(' . $tag . ') { media.first(' . $limit . ') { count, nodes { id, caption, code, comments.last(10) { count, nodes { id, created_at, text, user { id, profile_pic_url, username } } }, date, dimensions { height, width }, filter_name, display_src, id, is_video, likes { count }, owner { id, username, full_name, profile_pic_url }, thumbnail_src, video_url, location { name, id } }, page_info} }

raiym commented 7 years ago

almost done. please wait another hour or two

matkozikowski commented 7 years ago

@raiym So if you fix the problem we will able to get all images by tag ? thx.

raiym commented 7 years ago

Yes

On Mar 16, 2017 15:25, "Mateusz" notifications@github.com wrote:

@raiym https://github.com/raiym So if you fix the problem we will able to get all images by tag ? thx.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/postaddictme/instagram-php-scraper/issues/71#issuecomment-287041568, or mute the thread https://github.com/notifications/unsubscribe-auth/AFb0GRtBqjY33evRjbYNO1ePU2r10ohVks5rmSpDgaJpZM4Md2hb .

raiym commented 7 years ago

But login and password will be required

On Mar 16, 2017 15:27, "Raiymbek Kapishev" r.kapishev@gmail.com wrote:

Yes

On Mar 16, 2017 15:25, "Mateusz" notifications@github.com wrote:

@raiym https://github.com/raiym So if you fix the problem we will able to get all images by tag ? thx.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/postaddictme/instagram-php-scraper/issues/71#issuecomment-287041568, or mute the thread https://github.com/notifications/unsubscribe-auth/AFb0GRtBqjY33evRjbYNO1ePU2r10ohVks5rmSpDgaJpZM4Md2hb .

matkozikowski commented 7 years ago

Thats fine. After fix it please let us know what we should change :)

aik27 commented 7 years ago

@raiym, really, don't rush with login and password. Some commercial plugins works without authorization. Just emulate browser request, get csrftoken cookie and grab tags and location data by http://instagram.com/query/ endpoint.

raiym commented 7 years ago

Thank you guys. I already have done it.

https://github.com/postaddictme/instagram-php-scraper/releases/tag/v0.5.0

I am planning to add session caching (because now every time php dies login required). Maybe for now you will cache sessionId on your own

pinotarantino commented 7 years ago

Thank you veruy much raym, very good job, just installed now! It works fine but with a little change in my case:

Thank you again! ;)

jetonr commented 7 years ago

I created a dummy instagram account and got only the sessionId from the coockie and it works when I send the request. My question is will this work when making multiple requests from different domains and do they ever expire?

raiym commented 7 years ago

@jetonr i think it will work, but because Instagram links session for ip and browser (I think)you need to perform login on each server But I suggest to use one account only from one ip.