search

dilame / instagram-private-api

NodeJS Instagram private API SDK. Written in TypeScript.
MIT License
5.92k stars 1.14k forks source link

Add logging_client_events #1140

Open bolds07 opened 4 years ago

bolds07 commented 4 years ago

Notes

This request is not associated with any specifific function This is a periodical request of instagram app at any version which contains a bunch of logs of users interaction with the app and a validation code. I've looked on github many instagram apis and so far nobody was able (or interested) in reverse engineer it. I do have a little suspect that it can be related to other problems i've in my project. Does anyoe have the skills to reverse it?

i'm pasting here a sample request of what im talking about

/logging_client_events HTTP/1.1
X-IG-Connection-Type: WIFI
X-IG-Capabilities: 3brTvw==
X-IG-App-ID: 567067343352427
User-Agent: Instagram 100.0.0.17.129 Android (26/8.0.0; 480dpi; 1080x1920; samsung; SM-A520F; a5y17lte; samsungexynos7880; pt_BR; 161478664)
Accept-Language: pt-BR, en-US
Content-Type: multipart/form-data; boundary=WclRaGE_-iaawkAGZnprHXDKaqrJ8r
Accept-Encoding: gzip, deflate
Host: graph.instagram.com
X-FB-HTTP-Engine: Liger
Connection: keep-alive
Content-Length: 1576

--WclRaGE_-iaawkAGZnprHXDKaqrJ8r
Content-Disposition: form-data; name="access_token"

567067343352427|f249176f09e26ce54212b472dbab8fa8
--WclRaGE_-iaawkAGZnprHXDKaqrJ8r
Content-Disposition: form-data; name="format"

json
--WclRaGE_-iaawkAGZnprHXDKaqrJ8r
Content-Disposition: form-data; name="cmsg"; filename="ab67f151-d0ae-4c63-a45e-3fa8eedf56ab_1_zero.batch.gz"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

?T?n?6?=GqQ?Mb??kg`???.
??(?%?Z??A????i&?>?E?z??????%h???<d]??P?#?    ?h??A ???9D???^?LT}?A
#?)??=S{??!a?`(a?CE)#?0?<
S?<N??r!?,?9??#I)?9?d???  NIH2???dy??lU?j[
12?,???*?RWRFD?8?\?,'T&.??9?Jw???q???
#;U????7  b?c`?\o=#????????TZ????7?!;\??T?T?WWm'??,????F?4?X?O0$?rF ??8?iK???#?k^x/?+OW+??=??q
?6?????z    JQ.????8b?Q?]???,??%?????]???????b??b?(?`F?SiW8-|3?-???r???r?:N?Y??ls?'H?`??7r^5????x?xV(?[?7ll?CR??hwI??????8?y1~?&????/???o?Fl.d?
;????c????n?????*??j???6G1?q`?2-?m?N<-_rT??~????G???_^???F???+g.u+????\?V?'v?j?QF?V?w????E?i+?c??9?\??u?????'???k??"? ?   F???K???+?`
?F:N8?????!??5??\??n????t??u
??|B???"??n???0}S1g?y???K7?q?????n??h?j???]er?O???t*??????Kr??4c?w?T?????????g????!!?QFd?????;?,?X?B/???b?P@ "???????????3?}6????O??^?v??5?-??dy?Lff??n?D?Fk?X-??\?????~4m?n\l??G????p??ek??q?C[??,0v?q???J(/???W
--WclRaGE_-iaawkAGZnprHXDKaqrJ8r
Content-Disposition: form-data; name="sent_time"

1566875183.401
--WclRaGE_-iaawkAGZnprHXDKaqrJ8r
Content-Disposition: form-data; name="cmethod"

deflate
--WclRaGE_-iaawkAGZnprHXDKaqrJ8r--
Nerixyz commented 4 years ago

Content-Disposition: form-data; name="cmsg"; filename="*.gz" Content-Type: application/octet-stream Content-Transfer-Encoding: binary

Content-Disposition: form-data; name="cmethod" deflate

The contetnt is deflated (the input starts with 0x78). You have to infalte it. I can't reproduce the multipart/form-data, for me it's only urlencoded. If you could state what you did in order to get this request it would be helpful. Maybe it's the version (you're using v100, I'm using v136). You can also send the data base64 compressed so it's decompressable without data-loss. (In Burp you can use [right-click] > Send to decoder and en-/decode it).

bolds07 commented 4 years ago

I can't reproduce the multipart/form-data, for me it's only urlencoded.

As the endpoint suggests, this is some sort of user log... it is send periodicaly from the instagram app without the need of any specific human interaction.

if the log is small the app simple urlencode a normal json as you said... if the log is larger then instagram compresses it and send this form...

In order to get this multpart form simple roam around with the app for 10 or 15 minutes... like some photos, search something... do any normal interaction with the app and then you will get it....

i do believe that the data thats is deflated is the very same urlencoded... THAT CONTAINS SOME SORT OF DATA VALIDATION CRYPTOGRAPHY which im not able to replicate, and i would like very much to know if anyone is

Nerixyz commented 4 years ago

simple roam around with the app for 10 or 15 minutes...

This isn't happening in v136 (or not that often). It's rather sent as json.

But I used v133 and there it still happens.

THAT CONTAINS SOME SORT OF DATA VALIDATION CRYPTOGRAPHY

That's not cryptography. It's deflated, you have to inflate it. It contains something like this:

{
    "seq": 1,
    "app_id": "567067343352427",
    "app_ver": "133.0.0.32.120",
    "build_num": "204019472",
    "device_id": "...",
    "family_device_id": "...",
    "session_id": "...",
    "uid": "0",
    "channel": "regular",
    "log_type": "client_event",
    "app_uid": "0",
    "config_version": "v2",
    "config_checksum": "...",
    "data": [
        {
            "name": "android_string_impressions",
            "time": "...",
            "module": "IgResourcesAnalyticsModule",
            "tags": 1,
            "extra": {
                "impressions": {
                    "0123456789": 2
                },
                "string_locale": "en_US",
                "pk": "0",
                "release_channel": "prod",
                "radio_type": "wifi-none"
            }
        }
    ]
}
bolds07 commented 4 years ago

I know the data is deflated, this is not the concern of the question

"config_checksum":"qs|c=51a78d39a32ed5fed29c6b3e55fc1beb&ts=1566858438&qv=v1&s=1595da72d9f415d8a7b8eff881b0b0cf",
   "data":[

   ]

the question How to calculate checksum??

this qs field name reminds me of another checksum field that is found during the REGISTRATION REQUEST, which i basically dont know anybody that was able to reverse engineer it.

Nerixyz commented 4 years ago

The config checksum is basically from the last response. I don't know why you're freaking out about this.

bolds07 commented 4 years ago

1- im not freaking out... i have a question and i came to a public space ask for help... i thought the github was for that 2- the goal of any "instagram private api" out there is to behave like real instagram app... NONE of those apis do implement the request to this endpoint and it exists since instagram v35 so to instagram is pretty easy to identify who is a bot or not... "device_id" too long without sending log...

3- did you try do what you said? because if i try to post the request you proposed or any other without a real checksum i get the response:

{
   "error":{
      "message":"Invalid OAuth access token signature.",
      "type":"OAuthException",
      "code":190,
      "fbtrace_id":"AfI6ADiLg3WjKuQAMwWQ6yI"
   }
}

that is clear instagram reject it.

im sorry if i didnt started this issue with a clear question and i missused the therm cryptography for checksum... im doing lots of stuff in paralel. but if you are able to help with this subject, please dont it

bolds07 commented 4 years ago

I will explain my situation and maybe you will understand why i want this: i have an android app that uses a java version of mgp api (which i personally translated from php to java during 2 years)...

The users of my app often are caught with the famous "action_blocked" message, although the app is not intended for spam, it does follow, comment and like stuff...

the point is at the exact same moment the users receive the "action_blocked" message in the app, if they go to the original instagram app they are able to follow, like and comment.

this is running under real android devices so this is not a ip/proxy related problem... instagram is flagging users device_id as spam... so any other device_id is able to comment under same ip/account.

well this is happening over a long time and i'm out of new ideas about how to handle it. one last shot is try to see if replacating this request it will make instagram more permisive.

Nerixyz commented 4 years ago

did you try do what you said?

Kind of. I didn't have a token, so it's null and I received 200 OK with this response:

{
  "checksum": "",
  "config": "",
  "config_owner_id": "",
  "app_data": "{}"
}

But looking at the requests, the last checksum that was sent is used in the next requests. This is also visible in the code:

// v.133
// on save:
public final void 0Uj.BGe(InputStream p0,boolean p1)    //method@16aa
{
   // ...
   if (p0) {    
      0oW v0 = this.A00.createParser(p0);
      v0.null_nextToken();
      if ((v6 = 0XI.parseFromJson(v0)) && (v0_1 = v6.checksum) && !v0_1.isEmpty()) {    
         String checksum = v6.checksum;
         Map mainConfigHahmap = v6.mainConfigHashmap;
         Set blacklist = v6.blacklist;
         String configOwnerId = v6.configOwnerId;
         0XX v4 = 0XW.A01(0XW.A00());
         13c.A02(v4.A02(configOwnerId).A07());
         v4.A02(configOwnerId).A07().A01 = 1;
         String targetKey = "__config_checksum__";
         13c.A02(v4.A02(configOwnerId).A07());
         if (!checksum) {   
            13d.notNullOrThrow(targetKey);
            v4.A02(configOwnerId).A07().A00.put(targetKey, 13d.emptyObj);
         }else {    
            13d.notNullOrThrow(targetKey);
            v4.A02(configOwnerId).A07().A00.put(targetKey, checksum);
         }  
         // ....
}
// on load:
public final String 0XX.A03(String p0)  //method@1862
{
   13d config = this.getSomePigionSamplingPolicy(p0);
   String ret = null;
   13d.A01(config);
   Object v1 = config.A02;
   _monitor_enter(v1);
   if ((String checkSum = config.A04.get("__config_checksum__"))) { 
      ret = checkSum;
   }    
   _monitor_exit(v1);
   return ret;
}
bugsel commented 4 years ago

i think checksum comes in the response of the first request with a new device_id (~100k)

mmadymarov commented 4 years ago

In the latest uptade(142.0.0.34.110), no more deflated json in logging_client_events api, What do you think why instagram did this?

Nerixyz commented 4 years ago

@mmadymarov The Http requests are already compressed. Then there's no need to add another layer of compression.

caffeinum commented 4 years ago

I am working replicating this right now, if anyone's interested in synchronising work, we can connect via telegram or facebook messenger (same username as here)

bolds07 commented 4 years ago

I am working replicating this right now, if anyone's interested in synchronising work, we can connect via telegram or facebook messenger (same username as here)

do you have a repo? are you pushing here? i'm interested but i have zero progress on this

caffeinum commented 4 years ago

@bolds07 no, no repo yet. Currently have few examples collected, probably it'll help to collect more

Edit: this repo looks interesting, though: https://github.com/DamianoMagrini/instaverse/blob/857273e475a868585c91c07f626cbba5553ccb73/reversed-api/src/instagram/9568347_events.ts

Nerixyz commented 4 years ago

@bolds07 no, no repo yet. Currently have few examples collected, probably it'll help to collect more

Edit: this repo looks interesting, though: https://github.com/DamianoMagrini/instaverse/blob/857273e475a868585c91c07f626cbba5553ccb73/reversed-api/src/instagram/9568347_events.ts

This is for the web api.

350d commented 3 years ago

1- im not freaking out... i have a question and i came to a public space ask for help... i thought the github was for that 2- the goal of any "instagram private api" out there is to behave like real instagram app... NONE of those apis do implement the request to this endpoint and it exists since instagram v35 so to instagram is pretty easy to identify who is a bot or not... "device_id" too long without sending log...

3- did you try do what you said? because if i try to post the request you proposed or any other without a real checksum i get the response:

{
   "error":{
      "message":"Invalid OAuth access token signature.",
      "type":"OAuthException",
      "code":190,
      "fbtrace_id":"AfI6ADiLg3WjKuQAMwWQ6yI"
   }
}

that is clear instagram reject it.

im sorry if i didnt started this issue with a clear question and i missused the therm cryptography for checksum... im doing lots of stuff in paralel. but if you are able to help with this subject, please dont it

Make sure you send your data to URL without strain slash! That was my issue! Should be https://graph.instagram.com/logging_client_events and not https://graph.instagram.com/logging_client_events/

bolds07 commented 3 years ago

@350d @Nerixyz @caffeinum

do you guys have anything on it... i'm starting to map the events and would be good to join efforts

thisisprivateyo commented 3 years ago

Content-Disposition: form-data; name="cmsg"; filename="*.gz" Content-Type: application/octet-stream Content-Transfer-Encoding: binary Content-Disposition: form-data; name="cmethod" deflate

The contetnt is deflated (the input starts with 0x78). You have to infalte it. I can't reproduce the multipart/form-data, for me it's only urlencoded. If you could state what you did in order to get this request it would be helpful. Maybe it's the version (you're using v100, I'm using v136). You can also send the data base64 compressed so it's decompressable without data-loss. (In Burp you can use [right-click] > Send to decoder and en-/decode it).

How do you inflate the content? I've tried to decode it using burpsuit but nothing is happening. I've successfully decoded some other responses using it though

bolds07 commented 3 years ago

burp doesn't inflate, it only deflates... (at least the free version)

you must use burp and encode it as base64, then create a program which reads base64, decode then deflate... there are a few sites there that do it but is easier simple make your own

you cant simple control c + control + v because the inflated code has to much non ascii characteres so the copy paste process lose bunch of information...

brutalsuperman commented 3 years ago

I know the data is deflated, this is not the concern of the question

"config_checksum":"qs|c=51a78d39a32ed5fed29c6b3e55fc1beb&ts=1566858438&qv=v1&s=1595da72d9f415d8a7b8eff881b0b0cf",
   "data":[

   ]

the question How to calculate checksum??

this qs field name reminds me of another checksum field that is found during the REGISTRATION REQUEST, which i basically dont know anybody that was able to reverse engineer it.

I know the data is deflated, this is not the concern of the question

"config_checksum":"qs|c=51a78d39a32ed5fed29c6b3e55fc1beb&ts=1566858438&qv=v1&s=1595da72d9f415d8a7b8eff881b0b0cf",
   "data":[

   ]

the question How to calculate checksum??

this qs field name reminds me of another checksum field that is found during the REGISTRATION REQUEST, which i basically dont know anybody that was able to reverse engineer it.

If you still need it pm me

bolds07 commented 3 years ago

@brutalsuperman, this is github... there is no such a thing as "PM me"

either you have a solution for that and you can share here for the actual and future watchers of this topic if you somehow are looking for sell something, no thanks

thisisprivateyo commented 3 years ago

Qs field is for their native library called libquicksand. It's not soo hard to figure out

On Fri, 11 Jun 2021 at 15:08, Bolds @.***> wrote:

@brutalsuperman https://github.com/brutalsuperman, this is github... there is no such a thing as "PM me"

either you have a solution for that and you can share here for the actual and future watchers of this topic if you somehow are looking for sell something, no thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dilame/instagram-private-api/issues/1140#issuecomment-859608404, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNLMBKEWCNHZCLHYZFQAULTSIKG5ANCNFSM4MIYMSYQ .

crylonblue commented 3 years ago

Anyone knows, if instagram currently uses this for bot-detection?

crylonblue commented 3 years ago

https://github.com/DrunkSkipper/qs_stamp/blob/master/python/quicksand.py

found this implementation in python, anyone knows how to use this?

thisisprivateyo commented 3 years ago

They use safetynet now. Looking into quicksand is useless

On Thursday, 5 August 2021, crylonblue @.***> wrote:

https://github.com/DrunkSkipper/qs_stamp/blob/master/python/quicksand.py

found this implementation in python, anyone knows how to use this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dilame/instagram-private-api/issues/1140#issuecomment-893655388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNLMBJQJPZT5WDQE7PP2VLT3LEBJANCNFSM4MIYMSYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

crylonblue commented 3 years ago

I dont think they do so on ios. From what i saw on the ios app so far (197.0.0.20.119). They have two channels for logging_client_events "regular" and "micro-batch". The regular one does not have the config checksum, the micro-batch has.

crylonblue commented 3 years ago

To maybe solve this issue, the channel "regular" has no qs token. You can just send your events there. A simple implementation in python since, i am not that good in typescript.

import base64
import zlib
import requests

def logging_client_events(message_dict):
    data = {
        "format": "json",
        "access_token": constants.APP_ID + "|" + constants.ACCESS_TOKEN,
        "sent_time": self.api.get_client_time(),
        "system_uptime": self.get_system_uptime(),
        "compressed": "1",
        "messsage": base64.b64encode(zlib.compress(bytes(json.dumps(message_dict).encode()), 1)).decode()
    }

    log_request = self.session.post("https://graph.instagram.com/logging_client_events", data=data)

    return log_request

So i think its pretty much solved, now all you have todo is to add logging_client_events in each api_call, that has a event involved for example "ig_main_feed_request_succeeded" etc.

iPurya commented 2 years ago

@crylonblue what data we should send logging_client_event ? i know that json data but how to do this and stay undetected ?

kingbotss commented 1 year ago

I was able to test for video views, #1624

jamesbacarni commented 1 year ago

I was able to test for video views, #1624

Which data = [] did you use in the message to achieve this. I tried to replicate it, and the response is as always the same but the view count also stays the same.

I used the following events: instagram_organic_impression instagram_organic_sub_impression instagram_video_viewability_changed video_should_start video_buffering_started video_fetched video_format_changed video_started_playing video_buffering_finished video_played_time video_exited instagram_organic_viewed_impression instagram_organic_time_spent instagram_organic_vpvd_imp

crylonblue commented 1 year ago

I think some of the events get sent via MQTT instead of HTTP now. (But not sure)

DRVR1 commented 6 months ago

ACCESS_TOKEN

To maybe solve this issue, the channel "regular" has no qs token. You can just send your events there. A simple implementation in python since, i am not that good in typescript.

import base64
import zlib
import requests

def logging_client_events(message_dict):
    data = {
        "format": "json",
        "access_token": constants.APP_ID + "|" + constants.ACCESS_TOKEN,
        "sent_time": self.api.get_client_time(),
        "system_uptime": self.get_system_uptime(),
        "compressed": "1",
        "messsage": base64.b64encode(zlib.compress(bytes(json.dumps(message_dict).encode()), 1)).decode()
    }

    log_request = self.session.post("https://graph.instagram.com/logging_client_events", data=data)

    return log_request

So i think its pretty much solved, now all you have todo is to add logging_client_events in each api_call, that has a event involved for example "ig_main_feed_request_succeeded" etc.

hi, where do you get the constants.ACCESS_TOKEN? i don't find it even in burpsuite, is it account or hardware related? can i just randomize it? thanks.

brunoaduarte commented 6 months ago

ACCESS_TOKEN

To maybe solve this issue, the channel "regular" has no qs token. You can just send your events there. A simple implementation in python since, i am not that good in typescript.

import base64
import zlib
import requests

def logging_client_events(message_dict):
    data = {
        "format": "json",
        "access_token": constants.APP_ID + "|" + constants.ACCESS_TOKEN,
        "sent_time": self.api.get_client_time(),
        "system_uptime": self.get_system_uptime(),
        "compressed": "1",
        "messsage": base64.b64encode(zlib.compress(bytes(json.dumps(message_dict).encode()), 1)).decode()
    }

    log_request = self.session.post("https://graph.instagram.com/logging_client_events", data=data)

    return log_request

So i think its pretty much solved, now all you have todo is to add logging_client_events in each api_call, that has a event involved for example "ig_main_feed_request_succeeded" etc.

hi, where do you get the constants.ACCESS_TOKEN? i don't find it even in burpsuite, is it account or hardware related? can i just randomize it? thanks.

ACCESS_TOKEN is a fixed key for Instagram, you can hardcode both app_id and access_token in your application

567067343352427|f249176f09e26ce54212b472dbab8fa8

constants.APP_ID = 567067343352427
constants.ACCESS_TOKEN = f249176f09e26ce54212b472dbab8fa8

List of AppId and AppSecret of Facebook Mobile Apps

For developers: a list of app id and app secret from fbsh mobile devices. If you know any other ones, please add them in comments.

438142079694454|fc0a7caa49b192f64f6f5a6d9643bb28 - Ads Manager Android
350685531728|62f8ce9f74b12f84c123cc23437a4a32 - Facebook Android
6628568379|c1e620fa708a1d5696fb991c1bde5662 - Facebook iPhone
1479723375646806|afb3e4a6d8b868314cc843c21eebc6ae - Ads Manager App for iOS
1217981644879628|65a937f07619e8d4dce239c462a447ce - Instagram Web
567067343352427|f249176f09e26ce54212b472dbab8fa8 - Instagram Android(White Hat Edition)
alar1 commented 4 months ago

I was able to test for video views, #1624

Which data = [] did you use in the message to achieve this. I tried to replicate it, and the response is as always the same but the view count also stays the same.

I used the following events: instagram_organic_impression instagram_organic_sub_impression instagram_video_viewability_changed video_should_start video_buffering_started video_fetched video_format_changed video_started_playing video_buffering_finished video_played_time video_exited instagram_organic_viewed_impression instagram_organic_time_spent instagram_organic_vpvd_imp

Have you had any success today?