search

matamorphosis / Scrummage

A Holistic OSINT and Threat Hunting Platform
GNU General Public License v3.0
506 stars 81 forks source link

Instagram task not work #49

Closed Nicola966 closed 1 year ago

Nicola966 commented 2 years ago

Hi, all instagram task don't work. Why?

matamorphosis commented 2 years ago

@Nicola966 the instagram plugin has a known issue. This has to do with changes Facebook made a few months ago which mean the library used for searching for data no longer works. I'll leave this task open until the project comes to a conclusion, about whether we're able to find a resolution for this issue, or if we'll have to decommission the plugin.

Will keep you updated.

matamorphosis commented 2 years ago

Update

Unfortunately we are no longer able to provide unauthenticated search functionality for Instagram. Meta has recently cracked down on the way bots interact with their site. As far as obfuscating the Scrummage bot goes, we have a lot of inbuilt technology. This includes setting the user-agent to that a recent Firefox release. They recently made changes to check if a browser is interacting with Instagram, through JavaScript execution, so basically any use of the requests or the urllib python3 libraries do not work.

I explored using Selenium, which emulates legitimate browser activity, which was able to obtain information, but only a few times before becoming blocked. Thus, this is not a sustainable option.

For a long time the project has avoided going to the great lengths required to gain access to Facebook's GraphQL API, now called the Meta GraphQL API, which gives us the ability to search for Facebook and Instagram data. The reason being we don't provide the ability for users of the platform to use our API key, so you would have to register your own API key with Meta.

As we are now forced to resort to this, I am currently verifying myself with Meta for the purpose of R&D to develop plugins for both Facebook and Instagram; however, the success of this comes down to the response from Meta. If this is successful Instagram and Facebook search functionality will be provided, but as mentioned, you'll have to obtain an API key to use it.