AdguardTeam / DisableAMP

Disable AMP userscript
GNU Lesser General Public License v3.0
96 stars 6 forks source link

Doesn’t work on Google News (news.google.com) #88

Closed yetisyny closed 3 months ago

yetisyny commented 6 months ago

Google News (news.google.com) currently uses AMP, and currently this script doesn’t remove AMP from Google News links. However, I discovered a partial fix. Just add the following line

// @include      https://news.google.*/*

to the include section of the user script file right under the line that says

// @include      https://www.google.*/*

And then it will (sort of) work. But it doesn’t entirely fix the problem of AMP on Google News, it is just a partial fix.

To reproduce the problem of AMP being used on Google News, go to news.google.com in Firefox for Android with Tampermonkey and this script installed, and you will see the script not actually disabling AMP for Google News URLs. The URLs they use for AMP are formatted a little differently. A workaround I found is hitting refresh to get redirected to the actual news site and original article.

If you combine the earlier partial fix with refreshing the page, those 2 together seem to pretty much do the trick and remove AMP from Google News pages. Almost all the pages on Google News use AMP so this issue is easy to verify and test.

stanislav-atr commented 6 months ago

@yetisyny, hi, thank you for you feedback! Could you please also provide some urls and directions on what links has amp, as i wasn't able to reproduce this on my devices.

You also mentioned AMP urls being formatted differently, in case i wont be able to reproduce this, please, provide some examples of such urls.

yetisyny commented 6 months ago

OK, so if I go to news.google.com on Firefox for Android, it takes me to the specific URL https://news.google.com/foryou?hl=en-US&gl=US&ceid=US:en probably because I am logged into my Google account and in the en-US locale.

I am using your Disable AMP Beta user script in TamperMonkey for Firefox for Android.

Now Google News shows different stories to different people and it also regularly updates with new ones but I guess I can share what I get right now.

https://news.google.com/articles/CBMifmh0dHBzOi8vd3d3Lm5iY25ld3MuY29tL3BvbGl0aWNzL2RvbmFsZC10cnVtcC9hcHBlYWxzLWNvdXJ0LXdlaWdocy10cnVtcHMtaW1tdW5pdHktY2xhaW0tZWxlY3Rpb24taW50ZXJmZXJlbmNlLWNhc2UtcmNuYTEzMjI4MdIBK2h0dHBzOi8vd3d3Lm5iY25ld3MuY29tL25ld3MvYW1wL3JjbmExMzIyODE?hl=en-US&gl=US&ceid=US%3Aen - Judges skeptical of Trump’s immunity appeal at court hearing for 2020 election interference case on nbcnews.com, original URL is https://www.nbcnews.com/news/amp/rcna132281 and non-amp version is at https://www.nbcnews.com/politics/donald-trump/appeals-court-weighs-trumps-immunity-claim-election-interference-case-rcna132281

https://news.google.com/articles/CBMicWh0dHBzOi8vdGhlaGlsbC5jb20vcG9saWN5L2hlYWx0aGNhcmUvNDM5NTU0MC1zYW5kZXJzLWRlbW9jcmF0cy1sYXVuY2gtaW52ZXN0aWdhdGlvbi1pbnRvLWFzdGhtYS1pbmhhbGVyLXByaWNpbmcv0gF1aHR0cHM6Ly90aGVoaWxsLmNvbS9wb2xpY3kvaGVhbHRoY2FyZS80Mzk1NTQwLXNhbmRlcnMtZGVtb2NyYXRzLWxhdW5jaC1pbnZlc3RpZ2F0aW9uLWludG8tYXN0aG1hLWluaGFsZXItcHJpY2luZy9hbXAv?hl=en-US&gl=US&ceid=US%3Aen - Sanders, Democrats launch investigation into asthma inhaler pricing on thehill.com, original URL is https://thehill.com/policy/healthcare/4395540-sanders-democrats-launch-investigation-into-asthma-inhaler-pricing/amp/ and non-amp version is at https://thehill.com/policy/healthcare/4395540-sanders-democrats-launch-investigation-into-asthma-inhaler-pricing/

https://news.google.com/articles/CBMiLGh0dHBzOi8vd3d3LmJiYy5jb20vbmV3cy93b3JsZC1hc2lhLTY3OTA5MDYx0gEwaHR0cHM6Ly93d3cuYmJjLmNvbS9uZXdzL3dvcmxkLWFzaWEtNjc5MDkwNjEuYW1w?hl=en-US&gl=US&ceid=US%3Aen - India summons Maldives envoy over ministers' remarks on bbc.com, original URL is https://www.bbc.com/news/world-asia-67909061.amp and non-amp version is at https://www.bbc.com/news/world-asia-67909061

https://news.google.com/articles/CBMiZmh0dHBzOi8vd3d3Lm5iY25ld3MuY29tL25ld3MvdXMtbmV3cy9sYXN0LWJhdGNoLXVuc2VhbGVkLWplZmZyZXktZXBzdGVpbi1kb2N1bWVudHMtcmVsZWFzZWQtcmNuYTEzMjkzNtIBK2h0dHBzOi8vd3d3Lm5iY25ld3MuY29tL25ld3MvYW1wL3JjbmExMzI5MzY?hl=en-US&gl=US&ceid=US%3Aen - Last batch of unsealed Jeffrey Epstein documents released from nbcnews.com, original URL is https://www.nbcnews.com/news/amp/rcna132936 and non-amp version is at https://www.nbcnews.com/news/us-news/last-batch-unsealed-jeffrey-epstein-documents-released-rcna132936

But then there are a bunch not hosted on news.google.com, where a news.google.com URL redirects to the original news website. For example, these both quickly redirect from news.google.com to non-AMP sites and I only see the original news.google.com URLs for a fraction of a second before these redirect:

https://apnews.com/article/defense-secretary-lloyd-austin-illness-c52cfefc95924bc4725e19250fa424b8 - Defense Secretary Austin was treated for prostate cancer and a urinary tract infection, doctors say from apnews.com, doesn’t seem to be an AMP page

https://www.independent.co.uk/news/world/europe/facemasks-mandatory-spain-hospitals-b2475563.html - European country brings back mandatory facemasks in hospitals as Covid spikes again from independent.co.uk, doesn’t seem to be AMP page

It seems like maybe about 2/3 of news.google.com links redirect to non-AMP URLs and about 1/3 are AMP URLs for me currently. I don’t think this script needs to handle the non-AMP ones, just the AMP ones. Since its purpose is to bypass AMP to get to the original sites.

I hope this was helpful. You can probably duplicate it yourself if you use Firefox for Android and its TamperMonkey extension. I am sure you will probably get similar results if you go to news.google.com on mobile, getting AMP sites like these.

stanislav-atr commented 5 months ago

@yetisyny, i think i've found problematic links, fix is on the way.

stanislav-atr commented 5 months ago

@yetisyny, could you please check this userscript out? Looking forward to your feedback. disable-amp.user.js.zip

yetisyny commented 5 months ago

@yetisyny, could you please check this userscript out? Looking forward to your feedback. disable-amp.user.js.zip

I still get problems with certain links on news.google.com.

For example:

https://news.google.com/articles/CBMiM2h0dHBzOi8vd3d3LmJiYy5jb20vbmV3cy93b3JsZC1taWRkbGUtZWFzdC02Nzk0NDkwM9IBN2h0dHBzOi8vd3d3LmJiYy5jb20vbmV3cy93b3JsZC1taWRkbGUtZWFzdC02Nzk0NDkwMy5hbXA?hl=en-US&gl=US&ceid=US%3Aen - Israel says South Africa distorting the truth in ICJ genocide case - original URL is https://www.bbc.com/news/world-middle-east-67944903.amp which is also an AMP site. I searched the BBC site to find the original non-AMP page which is at https://www.bbc.com/news/world-middle-east-67944903 and seems to be that same URL except without the amp part at the end.

That was just the 2nd headline I found on the current news.google.com while logged in. The first one was a story at CNN which got redirected to the CNN website to a non-AMP page.

I did notice this user script working with some other pages though. For example the CNN page

https://www.cnn.com/2024/01/12/middleeast/weapons-us-uk-using-to-strike-yemens-houthis-intl-hnk-ml/index.html - What kind of weapons are the US and UK using to strike Yemen’s Houthis? - This news page, I saw it open an AMP page but the user script correctly redirected it to the original non-AMP page.

So it seems this version of the user script works on some sites like CNN and not others like BBC News. But there are plenty of other sites it seems to not detect or work on, too.

For instance https://news.google.com/articles/CBMiZGh0dHBzOi8vdGhlaGlsbC5jb20vaG9tZW5ld3MvaG91c2UvNDQwNDE2Ni1jb25zZXJ2YXRpdmUtam9obnNvbi1zcGVuZGluZy1kZWFsLWZydXN0cmF0ZS1yZXB1YmxpY2Fucy_SAWhodHRwczovL3RoZWhpbGwuY29tL2hvbWVuZXdzL2hvdXNlLzQ0MDQxNjYtY29uc2VydmF0aXZlLWpvaG5zb24tc3BlbmRpbmctZGVhbC1mcnVzdHJhdGUtcmVwdWJsaWNhbnMvYW1wLw?hl=en-US&gl=US&ceid=US%3Aen is an article from thehill.com and I can open the original site at https://thehill.com/homenews/house/4404166-conservative-johnson-spending-deal-frustrate-republicans/ using the interface at the top of the AMP site on Google News. Unlike BBC News, the original site for thehill.com doesn’t seem to be an AMP URL so this is the original article on that website.

Anyway it seems there are plenty of news websites whose stories are hosted as AMP pages at news.google.com with options at the top to link to the original website. This script seems to catch and redirect for some sites such as CNN but not others such as BBC and thehill.com.

Just as another example, this next one that is not redirected is from PBS Newshour, and Google News shows it at https://news.google.com/articles/CBMieWh0dHBzOi8vd3d3LnBicy5vcmcvbmV3c2hvdXIvc2hvdy93aGF0LXRoZS1uZXdseS1yZWxlYXNlZC1qZWZmcmV5LWVwc3RlaW4tZG9jdW1lbnRzLXJldmVhbC1hYm91dC1oaXMtc2V4LXRyYWZmaWNraW5nLXJpbmfSAX1odHRwczovL3d3dy5wYnMub3JnL25ld3Nob3VyL2FtcC9zaG93L3doYXQtdGhlLW5ld2x5LXJlbGVhc2VkLWplZmZyZXktZXBzdGVpbi1kb2N1bWVudHMtcmV2ZWFsLWFib3V0LWhpcy1zZXgtdHJhZmZpY2tpbmctcmluZw?hl=en-US&gl=US&ceid=US%3Aen and the article title is What the newly released Jeffrey Epstein documents reveal about his sex-trafficking ring. The original URL is https://www.pbs.org/newshour/show/what-the-newly-released-jeffrey-epstein-documents-reveal-about-his-sex-trafficking-ring . It seems in this case, there is an AMP site, too, on pbs.org, but the user script properly redirects from the AMP site on pbs.org to the original page on pbs.org, I think, because I saw a URL with “amp” in it quickly redirect to one without that. The AMP site is https://www.pbs.org/newshour/amp/what-the-newly-released-jeffrey-epstein-documents-reveal-about-his-sex-trafficking-ring, I think.

Anyway it looks like this user script has partially tackled this issue and seems to get sites out of AMP mode most of the time if you leave the news.google.com domain to look at the original website. But there also seems to be more to implement, namely automatically redirecting from the news.google.com - hosted webpages (which use AMP) to the original news websites. This specific aspect seems to just affect URLs directly hosted on news.google.com. But it seems this user script does a good job with news webpages hosted on the original news websites, getting the non-AMP versions of them, now.

Also it seems the news.google.com hosted AMP sites, they have a little menu thing at the top with a link symbol you can use to open the original news site. This opens it in a new browser tab, not the same one. So in the previous tab, the news.google.com hosted AMP sites stay open if you are manually opening the original sites. Ideally, these would automatically get redirected to the original sites, in the same tab, not a new tab, by this user script. The new tab thing is just how Google News has its website set up.

It looks like you have definitely made good progress in redirecting to non-AMP sites after loading the original websites in new tabs through the interface at the top of the AMP pages hosted by Google News, but there is still more to do, namely getting the AMP pages hosted at news.google.com to automatically redirect to the original websites.

Thanks for your work so far, I appreciate it.

stanislav-atr commented 5 months ago

@yetisyny, these links are not reproducible to me. What IPs do you use to access these links? To be clear, Disable AMP is not intended to be used with Google News app, and only works on the websites themselves.

yetisyny commented 5 months ago

I am using the Firefox for Android app, and browsing the website https://news.google.com, not using any other app like Google News. My IP address is just a normal United States IP address from the Internet Service Provider Spectrum (formerly Time Warner Cable). I noticed that this site had a lot of AMP pages on it. Another way to bypass the AMP pages is to switch Firefox for Android from mobile view to desktop view, which changes its user agent string, I think, but I prefer the mobile webpages because the desktop webpages are too zoomed out on mobile and their formatting is not mobile-friendly. I have an adblocker, uBlock Origin, and it is pretty good at blocking ads on the original news websites, but has trouble on the AMP versions hosted on news.google.com.

Anyway I am using Firefox for Android, NOT the Google News app, to avoid all the annoying ads in the Google News app. And I am using the script to bypass AMP because AMP webpages don’t work well with the adblocker and add an unnecessary annoying toolbar to the top of each webpage that clutters the view.

So I am using Firefox for Android to look at news on https://news.google.com and I have uBlock Origin and TamperMonkey installed and am using this script to bypass the AMP webpages so that the adblocking can work better. I hope that clears things up, that I am doing all of this in Firefox for Android with an adblocker on and the bypassing AMP pages thing is something I started trying to do to try to block more ads.

As far as my IP address, if you really want to know, it is 66.24.79.149. Not really sure why you need to know it but that is my current IP address, since you asked.

You can reproduce this just by going to https://news.google.com in Firefox for Android yourself. You don’t have to use the links I found. This is something anyone with Firefox for Android can reproduce by visiting the Google News website. I think the exact URLs Google generates might be temporary and generated for each user which might explain if they stop working later on.

I suggest you open https://news.google.com yourself in Firefox for Android and just click on random different stories you see, like you can just click the news links there starting from the top and go down one by one. That is probably the best way to reproduce this. That is how I reproduce this. I am sure you will encounter the same issue unless you have something else installed on Android to redirect you off the links hosted on news.google.com subdomain.

Anyway the only Android app I am using here is Firefox, not anything else. It is the only one I know of that can run user scripts such as this one. Presumably other users of this user script are also all using Firefox for Android too, since that seems like the only way to run user scripts on a mobile device that would get AMP websites served to it by Google websites.

Thank you.

stanislav-atr commented 5 months ago

@yetisyny, another script that should deal with amp embed disable-amp.user.js.zip

By the way, userscripts can be run with AdGuard for Android app.