Deobfuscating links in Direct Messages

[spodermenpls]

First of all, thanks for working on this addon and keeping it up to date with Twitter's bogus design changes, I appreciate the efforts! Unfortunately, I had to migrate now to the new theme, since Twitter phased the old layout out for good.

Your addon works fine with normal tweets, with messages though, it doesn't. I've read your explanation for it (use of React which doesn't show the texts and links in the source code). The t.co shortening crap in DMs really annoys the hell out of me, since I am exchanging website links with a developer back and forth due to beta testing efforts all the time, and need to copy the original ones, not the cloaked t.co's. Do you know of any possible way to potentially analyze these React modules, to deobfuscate the t.co-links there too? Or are those ReactJS things just generally too watertight?

And on a side note: You wrote in the repo documentation "In order to report bugs or ask for support please use Twitter Link Deobfuscator's GitHub page instead" - which every user visiting this repo already is, maybe adjust that one to avoid confused users. 😀

[theAlinP]

I'm inclined to try it. I'll see what I can do about it.

That line about the GitHub page is part of the README.md file, which can be found in the repository and the Firefox extension. I wrote it for anyone who reads the file, either because they cloned the repository and read the local copy or they extracted it from the .xpi file. It just happens that GitHub reads the contents of the README.md file and show it to the visitors which is why clicking the link would just refresh the page. I guess I can change that link to point to the Issues page. That should do it, right?

[spodermenpls]

That's good to hear, hopefully there's some kind of "attack vector". 😄

Yes, that makes sense. I was confused when I read it, since some projects split the coding and issue management into separate repos, which isn't the case here, obviously. I know that communicating issues over AMO is unpractical, so refering to the Github project makes sense, using the direct link to the issue section would make it clearer.

[theAlinP]

I will edit README.md very soon. As for the cloaked links in direct messages, I'm afraid you will have to get used to them, at least for a while.

[theAlinP]

I was a bit busy in the last few days and I didn't have much spare time for this but now I have a little.

I took a look at an old conversation I had on Twitter with a company's support team and I noticed that the links' original destinations are embedded in the HTML code which means that the links can be cleaned. I'll keep you posted in the following days with any progress I make.

[theAlinP]

OK, I managed to update the code to clean the links inside messages. It could clean the past messages successfully but I am unsure that it can clean new messages, loaded dynamically.

I haven't found a way to create Twitter test accounts/users like Facebook allows, I don't want to start a conversation with someone who follows me just to test the add-on and I certainly don't want to create a fake account, so would you and your colleague(s) be willing to test the changes?

[spodermenpls]

That's some great news! Of course I am willing to test, will you prepare an .xpi, or push it out directly on AMO?

[theAlinP]

I was thinking of a diff file with some basic changes to one of the scripts. I made the minimum amount of changes to that script just to see if it works and it's quite rudimentary. I won't create a commit right now, not in this state, so I can't update the add-on listed on addons.mozilla.org (AMO).

Here it is: TLD.diff.txt. I had to add the .txt extension because GitHub doesn't allow attaching .diff files.

How to use it:

1. I presume you already cloned the repository in the past, so you just have to pull the latest commits and checkout the master or the develop branch (commit 58a483f6ffee9603ba51e8b4cad2677688a09b6d on master and commit 15785dd37732cd5be7184b4531d4b143ced5087d on develop)
2. cd into the root of the repository and run git apply /path/to/TLD.diff.txt
3. In Firefox, open a new tab and in the address bar type "about:debugging" and hit Enter
4. Click "Load Temporary Add-on..." and when you are asked to select a manifest file or a package you have to navigate to the root of the repository and select the file "manifest.json" and click "Open". From this moment on and until you close the browser or remove the temporary add-on, the original one from AMO will be replaced by the temporary one.

Unfortunately, you will have to repeat the steps 3 and 4 every time you open your browser, at least until I update it through AMO.

[spodermenpls]

I am just a beta tester, not a developer (only my counterpart on Twitter), so I have no experience with Git and don't have it installed. I've created the "German locale" fork/commit/pull request solely on github.com. 😅

How about this, I've just forked this repo again and invited you as a collaborator, so you have write access to my repo fork and can apply the changes as they are intended, so afterwards I can simply download it and open it as a temporary extension (I am familiar with those), and your repo stays clean and dandy, until the script is ready?

[theAlinP]

OK, that's great.

[theAlinP]

I pushed one commit to the develop branch. Download it and see if it works.

[spodermenpls]

Thanks, I did that and observed the links in existing messages being converted properly, links in a new message that I've sent stay "t.co-ed", unfortunately.

Edit: My counterpart sent me a message with some links now, and those are getting uncloaked. Does Twitter (or your extension, respectively) handle "own" messages differently than received, "foreign" messages when displayed in the feed?

[theAlinP]

By "own" and "foreign" messages you mean the messages sent by you and the ones received?

[spodermenpls]

Yes.

[theAlinP]

I don't see why the extension would handle the 2 types of messages differently. Basically, it just looks for new children that are added to a <section> element and cleans the links inside them. I just took a quick look at the page source and I didn't see any differences between the 2 that would normally cause that. Anyway, I'll think more about this tomorrow.

[spodermenpls]

Shall I supply you with screenshots of the element inspector, while displaying the details of the sections that contain the cloaked/uncloaked links?

[theAlinP]

No, finding the cause by looking at screenshots would be too slow. Why don't you follow me on Twitter and then I'll be able to send direct messages to you. This way I will easily find out why the add-on can't clean the sent messages.

[spodermenpls]

Alright, I've thought maybe there might be some code differences, also since I tweak Twitter additionally with the GoodTwitter2 userscript + Violentmonkey (and other extensions as uBlock, uMatrix) which theoretically may have an effect on this. My DMs are open for anyone anyway, but I've followed you so you know which account is mine. 😉

[theAlinP]

I understand. Out of the 3, only GoodTwitter2 might conflict with the extension.

If you say that only the links from the messages that you receive are being uncloaked, all I need to do is to send you some messages with some links in them and tweak the extension progressively until it cleans the sent messages, too.

[theAlinP]

I updated the add-on to fix a small bug. This fix I already included in the last commit to the fork but it doesn't hurt to update the extension. This update also addresses the issue you mentioned earlier about the README.md file which contains a phrase that could be confusing.

[theAlinP]

Done!

I published a new version on addons.mozilla.org. You can update the add-on now.

[spodermenpls]

Superb!

I was surprised though by the newly requested "Access to browser tabs" permission (which didn't show up while beta-testing), can you summarize what makes it necessary? And is it constricted (like the other permission) just to "twitter.com"-tabs, or gives it - in theory - TLD now access to all browser tabs?

[theAlinP]

It needs it to search through the tabs and find the ones that access the twitter.com domain: https://github.com/theAlinP/twitter-link-deobfuscator/blob/b58204ec54f61888be3e060b2aad01acc2cdfbeb/scripts/background_script.js#L162

It selects the following tabs:

1. active: true => select all the tabs that are active (that have been used in the current session and have not been discarded)

2. url: "*://*.twitter.com/*" => select all the tabs with URLs having the following structure:

   • protocol: any (*)
   • subdomain: any (*)
   • domain: twitter.com
   • path: any (*)

Basically, it selects the tab/tabs that are active AND access twitter.com.

It then intercepts the network transfers to the tabs that fulfill the 2 conditions, uncloaks the Twitter Cards then forward the modified JSON files which are then parsed by the scripts on the page. It then updates the counters of each individual tab. As you have noticed by now, different tabs have different counters - or none. The Twitter Cards are uncloaked by background_script.js which intercepts the network transfers and the links are uncloaked by content_script.js which is injected into the page. There is other way to uncloak the Twitter Cards.

Here are more details about that permission request, here's more about the manifest.json permissions and about the tabs API. That permission gives it access only to the URL, title and icon of all the tabs, not their contents, not even in theory.

It can only modify the pages from the twitter.com domain due to the following lines:

1. inject scripts https://github.com/theAlinP/twitter-link-deobfuscator/blob/b58204ec54f61888be3e060b2aad01acc2cdfbeb/manifest.json#L13
2. intercept network requests https://github.com/theAlinP/twitter-link-deobfuscator/blob/b58204ec54f61888be3e060b2aad01acc2cdfbeb/manifest.json#L46

[theAlinP]

I'm not sure why Firefox didn't ask for that permission before, but the add-on has been accessing the tabs' URLs since the first time I updated it to intercept the network requests.

I guess the reason why Firefox doesn't ask for permissions when installing a temporary add-on is because this is done primarily by developers and testers and it would be ignored because it would only slow down the development.

[spodermenpls]

@theAlinP That's what I call textbook documentation, thank you for all the details! The new permission is therefore needed because of the necessity for network interceptions to uncloak the Cards, which wasn't implemented before. I see the whole picture now.

PS: By "There is other way to uncloak" you probably meant "There is no other way to uncloak", right? 😄

[theAlinP]

Yeah, I edited that comment so many times that I must've forgotten to add that word.