getThreadInfo and getThreadHistory NOT WORKING ANYMORE!

[Dudelmoser]

Hey guys,

I think starting yesterday getThreadHistory ceased to work. It returns an empty array for every thread. getThreadInfo now throws an error. It seems like the payload is empty which causes a "cannot read property '0' of undefined" error.

Could you do me a huuuge favor and look into this as soon as possible? I'm not familiar with the protocol facebook uses and my bachelor thesis is due on June 14th (I haven't really documented it yet with screen captures or so either, so my messenger is kinda worthless at the moment).

listen, getThreadList, getFriendsList, getThreadPictures etc. still seem to work.

Thanks in advance :)

[Dudelmoser]

By the way, you can link my messenger as soon as it works again if you want to. It has a pretty good looking material design based UI, almost all messenger features aswell as a custom meme generator, regex-based chatbot, custom autotext, themes and end-to-end encryption support.

[Schmavery]

@Dudelmoser Any chance this is related to https://github.com/Schmavery/facebook-chat-api/issues/486? Can we get a quick example of some non-working code so that we can test it on our end? We accept PRs adding links to related projects in the README 😄

[Dudelmoser]

You are right, it seems to be related to my account. It's still weird cause usually my account got blocked when I logged in too often or spammed requests due to bugs in my app (or even 24h banned when I tried to login with wrong credentials a few times). So the API failed to login in the first place and told me about being potentially banned. Now I can use the official messenger without any issues and even the unofficial API connects and works properly until I use a method related to thread_info.php.

Clone https://github.com/Dudelmoser/Mazenger, run npm install and npm start and go to localhost:3000 if you want to try my app. The docs are totally outdated etc., but I hope it works. If not, please tell me what doesn't cause I haven't done any deployment tests and the repository is currently just used as a backup until I'm done with my bachelor thesis.

Thanks again :)

[Dudelmoser]

By the way, didn't you say hosting the API online worked for others? The Heroku server I tried (an Irish one as far as I remember) got my accounts instantly banned. It's not that weird considering I'm from Germany but it's the closest Heroku server I could get. So it seems like connecting to FB via VPN will easily get you banned, too.

[Schmavery]

I'm pretty sure @bsansouci tried Mazenger an hour ago and it seemed to work fine. Maybe try with a different account for the purposes of getting some pictures etc for your thesis.

I'd be curious to know how we can fix our call to thread_info.php, but since this is only broken on your account, you'd need to help out with investigating that. If you log into facebook in the browser, open your dev tools and find a request to thread_info.php, maybe you can identify some different fields than what we're sending?

As far as services heroku are concerned, we've had mixed results. The main problem is that if you log in from a lot of different locations, facebook can get worried and will ask you to do some quick verification, which the api can't do. It's possible that you can work around this a little using getAppState, but it's not guaranteed.

[Dudelmoser]

I will investigate that further once I'm done with my thesis. I have one week left to write another 50 pages, so I try to avoid coding.

As a temporary fix you could check whether resData.payload.threads exists before assigned it's first value to threadData (getThreadInfo.js, line 33). Maybe log a hint about the function potentially not working correctly, but I haven't checked how that response looks like without restrictions.

The session cookie stays valid even after being banned, indeed. I haven't found out for how long though, feels kinda random.

[AstroCB]

I have been having this issue as well for about a week. All calls to getThreadInfo fail because the payload from the response data is empty.

[Schmavery]

@mangopearapples tested getThreadInfo/getThreadList recently as part of #488, so this must be affecting different people differently. In order to diagnose, we're going to need help from the people that are experiencing the problem.

[kieferlam]

@AstroCB It is still working on my end.

Maybe Facebook have made changes that are still being rolled out? Nothing has changed yet in the UK

[bsansouci]

I've seen this error on one of our bots. The reason is that it seems like FB updated their API so that getThreadInfo just plain doesn't work anymore. It does NOT return any thread information.

The quick fix is to revert to using getThreadList and query for the last 5 threads each time you get a message and pray that you don't get 6 messages all at once. That call will return the same data as getThreadInfo.

I'm not sure why FB did this change, getThreadInfo was pretty darn useful.

[sh5dow]

Hey guys, i have been experiencing issues with getThreadHistory. I get empty history for the all requests and all accounts i am trying it with, and i know, that the history is there. Does anyone have a hint, why its not working? Thanks for any help and take care

Code:

api.getThreadHistory(threadId, 50, null, (err, history) => {
        if(err) return console.error(err);

        console.log('history: ', history);
});

[kieferlam]

A couple of people (including me) think it's because Facebook are rolling out changes to the way they access history. Unfortunately (or fortunately for me :P), this means that the problem can only be diagnosed and fixed by people in the region where the new version is used.

It is still working fine for me which probably means my area isn't affected, but yours is.

[sh5dow]

Thanks for commenting out so soon. I will try to investigate it. I tried it a week or two ago and it worked flawlessly. Will post update on the issue, but if anyone comes with the solution, i will be happy to use the working one :D

[Schmavery]

Yeah, sorry about this @sh5dow. Any investigation you're able to do is hugely welcomed and let us know if we can help out in any way.

[kieferlam]

Welp. Within the past hour, I have also started experiencing this problem.

[kieferlam]

Looks like they moved to graph.

A post request to messenger.com/api/graphqlbatch/ with form data:

queries: {
    "o0": {
        "doc_id":"1386[REDACTED]407",
        "query_params": {
            "id" : "10[REDACTED]12",
            "message_limit" : 20,
            "load_messages" : 1,
            "load_read_receipts" : false,
            "before" : null
        }
    }
}

Will get me something like this:

{  
   "o0":{  
      "data":{  
         "message_thread":{  
            "thread_key":{  
               "thread_fbid":null,
               "other_user_id":"[REDACTED]",
               "legacy_thread_id":"mid.1473512063492:f8dde1ab2e38cffa56"
            },
            "name":null,
            "all_participants":{  
               "nodes":[  
                  {  
                     "messaging_actor":{  
                        "id":"[REDACTED]",
                        "__typename":"User"
                     }
                  },
                  {  
                     "messaging_actor":{  
                        "id":"[REDACTED]",
                        "__typename":"User"
                     }
                  }
               ]
            },
            "last_message":{  
               "nodes":[  
                  {  
                     "snippet":"SNIPPET REDACTED"
                     "message_sender":{  
                        "messaging_actor":{  
                           "id":"[REDACTED]"
                        }
                     },
                     "timestamp_precise":"1498160871220",
                     "commerce_message_type":null,
                     "extensible_attachment":null,
                     "sticker":null,
                     "blob_attachments":[  

                     ]
                  }
               ]
            },
            "unread_count":0,
            "messages_count":1274,
            "image":null,
            "updated_time_precise":"1498160871220",
            "mute_until":null,
            "is_pin_protected":false,
            "is_viewer_subscribed":true,
            "thread_queue_enabled":false,
            "folder":"INBOX",
            "has_viewer_archived":false,
            "cannot_reply_reason":null,
            "ephemeral_ttl_mode":0,
            "customization_info":{  
               "emoji":null,
               "participant_customizations":[  
                  {  
                     "participant_id":"[REDACTED]",
                     "nickname":"Lam kiev"
                  }
               ],
               "outgoing_bubble_color":null
            },
            "thread_admins":[  

            ],
            "approval_mode":0,
            "joinable_mode":{  
               "mode":"0",
               "link":""
            },
            "thread_queue_metadata":null,
            "event_reminders":{  
               "nodes":[  

               ]
            },
            "montage_thread":null,
            "last_read_receipt":{  
               "nodes":[  
                  {  
                     "timestamp_precise":"1498160871220"
                  }
               ]
            },
            "related_page_thread":null,
            "rtc_call_data":{  
               "call_state":"NO_ONGOING_CALL",
               "server_info_data":"",
               "initiator":null
            },
            "associated_object":null,
            "privacy_mode":1,
            "reactions_mute_mode":"REACTIONS_NOT_MUTED",
            "mentions_mute_mode":"MENTIONS_NOT_MUTED",
            "customization_enabled":true,
            "thread_type":"ONE_TO_ONE",
            "participant_add_mode_as_string":"ADD",
            "messages":{  
               "page_info":{  
                  "has_previous_page":true
               },
               "nodes":[  
                  {  
                     "__typename":"UserMessage",
                     "message_id":"mid.$cAAAAABc0vfdi-P-00Vcx7psPBTQ3",
                     "offline_threading_id":"6283065304897893431",
                     "message_sender":{  
                        "id":"[REDACTED]",
                        "email":"[REDACTED]\u0040facebook.com"
                     },
                     "ttl":0,
                     "timestamp_precise":"1497999502545",
                     "unread":false,
                     "is_sponsored":false,
                     "commerce_message_type":null,
                     "customizations":[  

                     ],
                     "tags_list":[  
                        "app_id:237759909591655",
                        "source:chat:orca",
                        "inbox"
                     ],
                     "platform_xmd_encoded":null,
                     "message_source_data":null,
                     "montage_reply_data":null,
                     "message_reactions":[  

                     ],
                     "message":{  
                        "text":"MESSAGE REDACTED"
                        "ranges":[  

                        ]
                     },
                     "extensible_attachment":null,
                     "sticker":null,
                     "blob_attachments":[  

                     ],
                     "snippet":"SNIPPET REDACTED"
                  },
                  {  
                     "__typename":"UserMessage",
                     "message_id":"mid.$cAAAAABc0vfdi-P_mcVcx7qWmzg1B",
                     "offline_threading_id":"6283065350394088769",
                     "message_sender":{  
                        "id":"[REDACTED]",
                        "email":"[REDACTED]\u0040facebook.com"
                     },
                     "ttl":0,
                     "timestamp_precise":"1497999515249",
                     "unread":false,
                     "is_sponsored":false,
                     "commerce_message_type":null,
                     "customizations":[  

                     ],
                     "tags_list":[  
                        "sent",
                        "source:chat:web",
                        "inbox"
                     ],
                     "platform_xmd_encoded":null,
                     "message_source_data":null,
                     "montage_reply_data":null,
                     "message_reactions":[  

                     ],
                     "message":{  
                        "text":"MESSAGE REDACTED"
                        "ranges":[  

                        ]
                     },
                     "extensible_attachment":null,
                     "sticker":null,
                     "blob_attachments":[  

                     ],
                     "snippet":"SNIPPET REDACTED"
                  }
               ]
            }
         }
      }
   }
}{  
   "successful_results":1,
   "error_results":0,
   "skipped_results":0
}

Sorry for a long comment. I've removed all but 2 messages so you can see the format of what's returned.

But there's some interesting data here. Most of the keys appear to be the same as before however they seem to be using strings for some of the values like thread_type being ONE_TO_ONE. I'd guess it's because they are using enumerations on their end.

[Schmavery]

This is awesome @mangopearapples, thanks a lot for sharing. A number of queries are switching to the graphql format and I'm still trying to get a picture of how we can use this reasonably (ie without just copy pasting some just query string). Is the string "o0" at all meaningful or can it be anything? Are you able to establish whether any of the elements in the query are constant?

[kieferlam]

No idea what the "o0" means but it's consistent when loading thread messages. Another request related to thread history, which appears to load thread images, has "q2" instead. These are consistent between ONE_TO_ONE chats and GROUP however they might not be consistent with other users so we need confirmation from someone else for that. Maybe it changes with time but I haven't done enough testing to know that yet.

The query format appears to be the same for both GROUP and ONE_TO_ONE thread types. I don't really have anyway to test with pages or events.

Loading more history is done in the same way as before as well.

{  
   "o0":{  
      "doc_id":"13.....7",
      "query_params":{  
         "id":"12.....0",
         "message_limit":21,
         "load_messages":1,
         "load_read_receipts":true,
         "before":1498118741268
      }
   }
}

This query loads the most recent 21 messages before the timestamp at "before".

[Schmavery]

What I'm wondering is if the doc_id encodes the query (so it will always be the same for the same request), and whether changing the "o0" in the request would have no effect other than changing the corresponding key in the response (allowing you to potentially make multiple queries in the same request). I may be totally off though.

I'm not sure what we can learn from this yet, but it looks like fbchat (the python version of this api) has already implemented support for this: https://github.com/carpedm20/fbchat/blob/master/fbchat/graphql.py

[sh5dow]

I made some research and went i tried the last version of API, it started working again, now i get mesasge history as i used to. Strange though.

[kieferlam]

Yup they started using thread_info.php for me again as well and looks like only one method is used at a time so I can't even see the graph method anymore.

@Schmavery Can't look into it anymore until they use graph again but doc_id was the same for all queries as far as I can tell. Maybe it changes with time or session though.

[Schmavery]

hmm ok, thanks for the help everyone, glad it's "working" now, we'll just have to see what happens next 😛

[bsansouci]

Hey @Dudelmoser @mangopearapples, I pushed on Saturday an update to master with two new functions getThreadHistoryGraphQL and getThreadInfoGraphQL. Those are probably going to be the replacement for the two old functions as we move towards a graphql world, but we'd like to test them a bit beforehand.

Would you mind trying them out?

The formatted data's slightly different from before though. The formatting functions are inlined in the file which should be pretty straightfoward to follow. I'll write docs soon.

[kieferlam]

@bsansouci Thanks for this. The graph functions are working great for me. I've only tested with one to one and group chats so far. Messages and attachments seem to work fine but formatAttachmentsGraphQLResponse is missing MessageAudio which is when you record audio directly to Messenger.

Here's the attachment dump:

{ __typename: 'MessageAudio',
       attribution_app: null,
       attribution_metadata: null,
       filename: 'audioclip-150******000-3**6.mp4',
       playable_url: 'url_to_mp4_file',
       playable_duration_in_ms: 3286,
       is_voicemail: false,
       audio_type: 'VOICE_MESSAGE',
       url_shimhash: 'a hash',
       url_skipshim: true }

I can do this in a PR though.

Haven't tested getThreadInfoGraphQL yet. The doc_id kind of worries me. It looks like it should be unique for every person or thread but mine was the same as the one in the form. But if it works then it works.

[Schmavery]

Thanks for testing, @mangopearapples. Our understanding is that the doc_id is unique to the query, as a kind of stored procedure where it will correspond to a query on the server. A PR to fix MessageAudio would be great, it might even solve #383 😄

Just a note that the signature of our graphql functions could still change slightly, we're still working out the details with the help of all you guys!

[adminy]

Still no fix for this no?

[Schmavery]

@adminy @bsansouci wrote getThreadHistoryGraphQL and getThreadInfoGraphQL functions, which some people are using, but unfortunately no one has had the time to tweak these for compatibility and replace the getThreadHistory/getThreadInfo functions with them. Let me know if you're interested in helping with that and I'm happy to give more details.

[adminy]

@Schmavery Sure, These New Functions which replace the old ones, do they provide the same functionality?

actually just tested them, they work great 💯 So I was thinking, but I don't know what the old one's functionality was like different to the new one, they look same to me.

[Schmavery]

@adminy That's the idea, though their responses differ in some small ways. This is the difference that would have to be reconciled when we replace getThreadHistory/getThreadInfo.

[adminy]

@Schmavery

I parsed messages object like so:

    this.api.getThreadHistoryGraphQL(threadID, 50, undefined, 
      function(messages) {
          console.log(JSON.parse(messages['res'].substring(0, messages['res'].indexOf("]}}}}}")+7))['o0']['data']['message_thread'])      
      })

I got the messages object I needed, but I actually wanted the message objects themselves to I wrote a small for loop:

    this.api.getThreadHistoryGraphQL(threadID, 50, undefined, 
      function(messages_data) {
        var messages = JSON.parse(messages_data['res'].substring(0, messages_data['res'].indexOf("]}}}}}")+7))['o0']['data']['message_thread'];
        for(let message in messages['messages']['nodes'])
         console.log(messages['messages']['nodes'][message])
      })

I am new to the API so I don't know what format the old getThreadHistory yielded but I am willing to convert it if you give me a template JSON :) 👍

Here is my final version of JSON breakdown for what I essentially need.

  pullHistory(threadID, count, timestamp, callback) {

    this.api.getThreadHistoryGraphQL(threadID, count, timestamp, 
      function(messages_data) {
        var messages = JSON.parse(messages_data['res'].substring(0, messages_data['res'].indexOf("]}}}}}")+7))['o0']['data']['message_thread'];
        //console.log(messages)
        let ThreadHistory = {
          threadUsers: [],
          messages: [],
          messages_count: messages.messages_count,
          unread_count: messages.unread_count,
          name: messages.name,
          last_mesage: messages.last_message.nodes[0]
        }
        for(let participant in messages['all_participants']['nodes'])
          ThreadHistory.threadUsers.push({
              id: messages['all_participants']['nodes'][participant]['messaging_actor']['id'],
              type: messages['all_participants']['nodes'][participant]['messaging_actor']['__typename']
          })

        for(let message in messages['messages']['nodes'])
          ThreadHistory.messages.push({ //message_source_data, message_reply_data ?! maybe later on
            type: messages['messages']['nodes'][message]['__typename'],
            id: messages['messages']['nodes'][message]['message_id'],
            senderID: messages['messages']['nodes'][message]['message_sender']['id'],
            senderEmail: messages['messages']['nodes'][message]['message_sender']['email'],
            unread: messages['messages']['nodes'][message]['unread'],
            timestamp: messages['messages']['nodes'][message]['timestamp_precise'],
            text: messages['messages']['nodes'][message]['message']['text'],
            reactions: messages['messages']['nodes'][message]['message_reactions']
          })

        if(ThreadHistory.messages.length > 0)
          ThreadHistory.prev = ThreadHistory.messages[0].timestamp;

        callback(ThreadHistory)
        // console.log(ThreadHistory)
      })

[adminy]

Where did you take the doc_id? Is it really unique for every account? if so then can't we produce our own at login ?

[adminy]

Any Official Fix for getThreadHistory? It does not seem hard at all, so why isn't it updated?

[ravkr]

@adminy getThreadHistory or getThreadHistoryGraphQL? the latter works just fine (at least for me)

if you have problems try to change doc_id to eg. 1714575961918512 (just extracted this from messenger.com requests, also works in my case)

[ravkr]

This should work now... Can we close it?

[ravkr]

@Schmavery one more issue to close

[Schmavery]

thanks all