Update built-in instagram rules after HTML changes

[darkred]

Hi there @tophf

I noticed today that due to HTML changes the instagram built-in rules no longer work anywhere, nor in timeline, nor in profiles or posts themselves. Additionally I gladly see that now 1440p images are available by default (therefore no need for using modified User-Agent, as Image Max URL script/extension does, see #41).

With this PR I made all needed changes in order the built-in rules to work again.

I am aware that you don't have an instagram account to test via the API (?__a=1 URLs), so here are some screenshots from Firefox's JSON viewer while doing various requests with ?__a=1 to help you check my suggested changes.

image post


video post


carousel image post

 

Lastly, I believe that the following part is no longer needed or fixable and should be removed.

_getEdge: shortcode => unsafeWindow._sharedData.entry_data.ProfilePage[0].items[0].user
   .edge_owner_to_timeline_media.edges.find(e => e.node.shortcode === shortcode).node,
},

Test URL: https://instagram.com/instagram

[tophf]

Great, but instagram still works for me with the old rules so maybe the new html is deployed gradually or you got randomly selected into an experimental group as a frequent user of the site. It might make sense to combine the rules for time being somehow, if you can, otherwise this may break instagram for some/many users...

[darkred]

I confirm that instagram still works when not having login. So, I believe the new html is available to logged in users only. But, I find it overall difficult to combine my changes with the existing code, because it's not only different object properties that are used, but there are also two new ternary if (which replace ||), the latter of which is nested. So, I think it's better to just wait until the html changes get to non logged-in users too (hopefully it will occur soon), and then merge.

[darkred]

Could it be possible to add a function that would detect whether the user has logged in to instagram or not? The way I found to check the above, is to check for a sessionid cookie between subdomain and domain (www.instagram.com --> instagram.com ): Instagram's subdomain for pages is www.instagram.com, and when you have logged in it creates/uses a non-empty cookie sessionid in the instagram.com domain.

Here is screenshot while in https://www.instagram.com (clicking lock icon in omnibox | Cookies):


This way we could add my suggested changes as separate variations of the existing rules, i.e. :

if (loggedIn !== true) {
  // existing rule
} else {
  // my suggested variation
}

  I found this related Stackoverflow answer which is unfortunately quite complicated. Is there an easier way? Or my idea is not worth the trouble?

[tophf]

That's conceptually what I proposed but I think it should be possible to detect by simply checking the current DOM structure.

[darkred]

I tested it and it's working ok. I believe it's fine now.

Just one thing, please:

Lastly, I believe that the following part is no longer needed or fixable and should be removed.

   _getEdge:  !(isLoggedIn) ?
     shortcode => unsafeWindow._sharedData.entry_data.ProfilePage[0].graphql.user :
     shortcode => unsafeWindow._sharedData.entry_data.ProfilePage[0].items[0].user
       .edge_owner_to_timeline_media.edges.find(e => e.node.shortcode === shortcode).node,

[tophf]

With these changes instagram no longer works for me.

[darkred]

~~That's because there's the following issue now: !!$('svg[aria-label="Home"]') is wrongly always true, i.e. even when not logged in, while !!document.querySelector('svg[aria-label="Home"]') is false as expected when not logged in. Shouldn't I use the latter, instead?~~

Also, I had to put that declaration there, because isLoggedIn is also called later at lines 1973, 1976, 1988, which are all inside main array declaration. That's why in my new commit I had to redeclare isLoggedIn locally twice.

[tophf]

$ is the same as document.querySelector and I've just verified that using the latter doesn't help anyway.

[tophf]

The problem is that you're not calling the function.

[darkred]

Thanks and sorry about that, I got confused while trying things in console.

With my new commit I think I do it right: I actually call the function: isLoggedIn() instead of isLoggedIn.

(I didn't change the last few lines because in there the two redeclarations of isLoggedIn are both a const, not a function).

But unfortunately the script is again not working on IG, and I can't tell what's wrong. I believe it should work.

[tophf]

Try debugging it by setting a breakpoint inside the rule in devtools or by adding debugger; in code. Then run line by line and inspect the variables.

[darkred]

Ok, so I tried putting a breakpoint in line 1564 dotDomain.endsWith('.instagram.com') && but I think the script in its current state unfortunately bypasses the rule body (i.e. from ((isLoggedIn = () => !!$('svg[aria-label="Home"]')) => ({ and on) altogether.

I see in Network devtool, that whatever post I may hover in e.g. http://instagram.com/instagram, it wrongly tries to fetch https://www.instagram.com/instagram/undefined (screenshot)

  I've also tried a simpler approach, to have isLoggedIn as a method (if I use the right word), but it doesn't work either:

      dotDomain.endsWith('.instagram.com') && {
        isLoggedIn: () => !!$('svg[aria-label="Home"]'),
        e: null,
        s: (m, node, rule) => {
          if (!(this.isLoggedIn())) { 
            // ... 
          }
        },
      }

[tophf]

You need to place the breakpoint inside the code for s.

[tophf]

BTW I've tried logging in and the old rules still work for me so svg[aria-label="Home"] can't be used to detect the new html.

[darkred]

Ok, I tried putting the breakpoint on every line inside s and unfortunately it never breaks. Also, here is what I'm getting in console when hovering any IG post: . Notice that the rule body is now displayed in console in yellow, i.e. there's no syntax highlighting of its various properties (e). So, maybe with the current syntax (I quote it below for reference) the rule becomes effectively "invalid" ?

      dotDomain.endsWith('.instagram.com') &&
      ((isLoggedIn = () => !!$('svg[aria-label="Home"]')) => ({
        // ...
      })()),

  And, to clarify this: With "new html" changes, I was only referring to the differences in the JSON responses of the IG API to ?__a=1 URL requests, i.e. some names are different (see my screenshots in the OP) and now 1440p images/larger video URLs are also served. There's no visible change in the page layout/DOM. Sorry if I wasn't clear enough about this or for the misunderstanding. I use the svg[aria-label="Home"] selector to detect whether the user has logged in to IG, hence the isLoggedIn() naming. The matching element is the "home" icon that (among others) only exists when you have logged in:

[tophf]

Indeed, the rule's last line is wrong, it should be }))(),

Anyway, find out how to make your devtools work with breakpoints because it's absolutely indispensable.

I was only referring to the differences in the JSON responses of the IG API to ?__a=1 URL requests

In that case you don't need to check the svg element because it doesn't tell anything. I have this element when I log in but and the old rules are still working for me.

[darkred]

Indeed, the rule's last line is wrong, it should be }))(),

Thank you. This change fixed the code. That was why the breakpoints weren't working at all. Regardless, I'll try to become more familiar with the debugger devtools, thanks.

 

In that case you don't need to check the svg element because it doesn't tell anything. I have this element when I log in but and the old rules are still working for me.

Summary of my new rules vs old rules with the various kinds of IG posts:

(Old rules)
Logged out:	single images (1080) ✔️	galleries ❌	videos ✔️	igtv ✔️
Logged in:	single images (1080) ✔️	galleries ❌	videos ✔️	igtv ✔️

(New rules)
Logged out:	single images ❌	galleries ❌	videos ❌	igtv ❌
Logged in:	single images (1440) ✔️	galleries (1440) ✔️	videos ✔️	igtv ✔️

My new rules only work when logged in. That's why with this PR:

• when not logged in to use the old rules, and
• when logged in, then use the new rules.

Am I missing something? What do you mean when saying that I don't need to check the svg element? It's a working criteria to detect whether the user is logged in.

[tophf]

I see. Indeed, galleries don't work with the old rules, but they also don't work with the new code either because ?__a=1 url returns a login page instead of json.

[darkred]

I have noticed 2 different things:

• when logged out, Instagram allows only 4 requests (of any kind, i.e. page loads or ?__a=1 requests, it's the same) within a short time - in the 5th attempt it starts redirecting any request to https://www.instagram.com/accounts/login/, i.e. you can no longer load any IG page until either some time passes, or you login, or get a different IP. To reproduce, get a new ip, and load e.g. https://www.instagram.com/instagram/ in an incognito window, then press Reload (F5), wait and repeat: it will load ok the first 4 times, but in the 5th time it will start redirecting.
• The script as it is (without the PR) works ok in 3 out 4 cases, (videos, igtv, single images) with 1 exception: galleries. BUT, with the PR, when logged out, it only works for galleries (!) , i.e. the opposite than expected ! Could you please help me understand why is this happening? I have tried all these days and can't find anything wrong. Maybe I'm using either isLoggedin() function or isLoggedIn variable in a wrong way?

And, what could be done to deal with this ?__a=1 limitation? Stop making ?__a=1 when logged out? Limit them to 4 and then wait for e.g. 10 minutes until next requests ?

[tophf]

I don't know yet how to fix this other than stop using __a altogether and just fetch the post's URL, then try to parse the html.

[tophf]

Aha, __a didn't work for me due to a bug in Chrome when "block all cookies by default" is set in browser options. I'm merging this PR as is now and will refactor it a bit afterwards.

Thanks!

[darkred]

I am always glad to contribute to this project! Thanks for the guidance.

[darkred]

@tophf I noticed that, after your refactor, in some profile pages, when mouseovering single-image posts, for most of them the displayed images are not the max-res ones .

e.g. opening this profile https://www.instagram.com/aoua.creations/ the popup for most of the posts posts shows the 1080 image instead of the larger (1440) one. For example, mouseovering the latest post shows the 1080x970 image while the max image is 1440x1293 .

[tophf]

That's because this PR fortuitously broke getEdge so MPIV wasn't using the internal data from the page, instead it always made a remote request, which produced a bigger image. From now on, MPIV will always make a network request.

[darkred]

Thanks a lot for the fast response and for fixing this!

[darkred]

@tophf I have noticed another issue that occurs after this PR: in some video posts, saving a video via d keyboard shortcut results a very long filename and wrong extension (.jpg) .

E.g. open https://www.instagram.com/instagram/ , mouseover the newest post and press d to download, results in this very long filename and wrong extension:

dzz8VAigCQzMsF0AozMzMzMzNGBJkYXNoX2Jhc2VsaW5lXzFfdjERAHX+BwA=&_nc_rid=43136d6df6&ccb=7-4&oe=620234E7&oh=00_AT9pyNUYZdveRT_r_KFA4UQ36sLiGRKgQh1XJncH6jJvXQ&_nc_sid=83d603.jpg

But, if instead of downloading, as you mouseover, if you rightclick twice and then press t (to open the video in a new tab), and rightclick on the video | Save, then it's saved with the correct name+ext:

273431051_709255997146991_5519447976516055109_n.mp4

As it's opened in a new tab, the URL is:

https://scontent.cdninstagram.com/v/t50.2886-16/273431051_709255997146991_5519447976516055109_n.mp4?efg=eyJ2ZW5jb2RlX3RhZyI6InZ0c192b2RfdXJsZ2VuLjcyMC5jbGlwcy5iYXNlbGluZSIsInFlX2dyb3VwcyI6IltcImlnX3dlYl9kZWxpdmVyeV92dHNfb3RmXCJdIn0&_nc_ht=instagram.fath2-1.fna.fbcdn.net&_nc_cat=111&_nc_ohc=0B4XOfWUyLUAX-ScFlw&edm=AABBvjUBAAAA&vs=656532712139411_3890462379&_nc_vs=HBksFQAYJEdBczZUQkJ2eDhPSEVJVUNBRVh3ZlB5d0JKbE1icV9FQUFBRhUAAsgBABUAGCRHS1Y0U2hDQjU1X3B6eFlRQVB0UlRKVzRERjhWYnFfRUFBQUYVAgLIAQAoABgAGwAVAAAm2OX92M%2Fdzz8VAigCQzMsF0AozMzMzMzNGBJkYXNoX2Jhc2VsaW5lXzFfdjERAHX%2BBwA%3D&_nc_rid=bc97049b5b&ccb=7-4&oe=620234E7&oh=00_AT9pyNUYZdveRT_r_KFA4UQ36sLiGRKgQh1XJncH6jJvXQ&_nc_sid=83d603

And I see that the used filename is the last part of the above URL.

This only occurs when logged in, and only with some videos.

I can't seem to find what's the cause. Could you please help with this?

[tophf]

Should be fixed in 3545f541.

[darkred]

Yes, it's fixed now, thank you very much!