search

nextcloud / news

:newspaper: RSS/Atom feed reader
https://apps.nextcloud.com/apps/news
GNU Affero General Public License v3.0
866 stars 186 forks source link

Twitter warning in articles (twitter not loaded...) #1374

Closed radoeka closed 2 years ago

radoeka commented 3 years ago

IMPORTANT

Read and tick the following checkbox after you have created the issue or place an x inside the brackets ;)

Explain the Problem

Quite some articles that are displayed so a message from twitter:

Twitter not loaded as you did not give permission. Click **here** to modify. ....

See screenshot. Screenshot_News

I would like to get rid of this message, but I don't have a clue how. The link marked here is not clickable. The page code does not show the note/warning from twitter. There are similar warnings when searching with e.g. Google. But only in the search results and not in the article it refers to. I did not find an article with a solution :(

The issue is shown in the Android Notes app as well as in the NC News web GUI.

I contacted the Android Notes app developer, without solution.

Steps to Reproduce

Explain what you did to encounter the issue

  1. Use a feed with articles that refer to twitter
  2. Read an article that refers to Twitter
  3. The warning from Twitter shows up in the article

System Information

Is there anyway to get rid of the warning from Twitter? Would it possible to make the link marked here in the article clickable? Could it be that backend must be enabled for Twitter? If so how?

anoymouserver commented 3 years ago

Do you have any example feed for us to check? None of my over 100 feeds has anything similar.

radoeka commented 3 years ago

Do you have any example feed for us to check?

Here is an example: https://www.metronieuws.nl/in-het-nieuws/feed/

The warning is in that feed as well:

<div class="social-media-no-consent-box" style="border: 1px solid #ccc; padding:10px; margin-bottom:15px;">Twitter wordt 
niet geladen omdat je geen toestemming hebt gegeven. <a href="javascript:Didomi.notice.show();">Klik hier</a> om het aan 
te passen.<br/><small>Wel toestemming gegeven maar niet getoond, herlaad de pagina.</small></div><blockquote 
class="twitter-tweet-no-consent" data-width="500" data-dnt="true">

image

It is referring to some Javascript: Didomi.notice.show()

In another feed I've, a twitter msg is referred to, and it seems visible: https://www.want.nl/feed/ image

In this working case the URL seems to be: <a href="https://twitter.com/PopBase/status/1395603103446179844?ref_src=twsrc%5Etfw">May 21, 2021</a></p>

The feed that does not work, I've in feedly and there the article is just working.

radoeka commented 3 years ago

Seems to be related to this: https://developers.didomi.io/cmp/web-sdk/consent-notice/configuration-by-url but how to configure this?

SMillerDev commented 3 years ago

Do you have full text enabled?

radoeka commented 3 years ago

Do you have full text enabled?

Where is this setting? I've: image

I know, it is in Dutch. Full Text would be 'volledige tekst' in Dutch. I don't see it listed... Perhaps the translation needs to be looked at.

Anyway, perhaps you can tell which setting (from the top of the menu) it is?

SMillerDev commented 3 years ago

It's a setting on the feed level. In the same menu where you would delete/rename a feed.

radoeka commented 3 years ago

I just found it, and enabled it: image

When would this have an effect? On newly retrieved entries, of also the already retrieved ones?

anoymouserver commented 3 years ago

It's not caused by the full text fetching, but the Twitter box is included in the feed itself. 🤨

A hacky solution would be to add a custom CSS for this feed, but since it's only a div, the following could potentially also hide other boxed elements in future articles. Furthermore it will only work for the webapp and not in any client.

.custom-metronieuws-nl .body > div[style*="border"] {
    display: none;
}
radoeka commented 3 years ago

@anoymouserver and @SMillerDev do you get the same behavior? I think it is because our feed cronjob/crawlers is not recognized as 'bot' or 'crawler'. By the feed website. I think it has to with Didomi’s CMP (Consent Management Platform). I referenced to this a few comments earlier.

See: https://developers.didomi.io/cmp/web-sdk/consent-notice/bots

According the information at this page: https://developers.didomi.io/cmp/web-sdk/consent-notice/configuration-by-url#specific-configuration Something could be done via URL configuration: {{YOUR_WEBSITE_URL}}?didomiConfig.notice.enable=false But for me that does not work. For example: https://www.metronieuws.nl/in-het-nieuws/feed/?didomiConfig_notice_enable=false

Is there something the feed website, could do to recognize our news crawler as crawler?

Or is the NC News (web) client able to allow the user to use <a href="javascript:Didomi.notice.show();">Klik hier</a>???

anoymouserver commented 3 years ago

I have no idea what the requirements are, that this site would detect NC News as a bot. The fetcher identifies itself with the user-agent "NextCloud-News/1.0".

No .. apart from security implications, should javascript be allowed in feed articles, it wouldn't work either in that case, as the Didomi functionality isn't included at all.

In my opinion this box shouldn't be in this news feed in the first place. It wouldn't work (also with no other feed reader) and it's also not recommended to include active content into feeds at all (https://validator.w3.org/feed/docs/warning/SecurityRisk.html).

Grotax commented 3 years ago

@David-Development this is maybe also interesting for you, I don't have any idea how we should deal with all the cookie banners and consent/tracking buttons in the future.

I guess one way would be to remove all filtering from the backend and just store the item as blob, leave it to the client to handle the code safely. Or maybe we need some more advanced filter system in news basically an ad-blocker that beautifies the feeds.

But both will require major changes to the way the app works.

David-Development commented 3 years ago

The feed that does not work, I've in feedly and there the article is just working.

@radoeka Do you know if feedly is using the same rss api? If they use another protocol that might explain why they are able to embed those twitter posts correctly..

@Grotax Thank you for chiming me in here! I had another look at the provided feed. To me it looks more like something that is wrong when the rss feed is generated.. right? I mean if you open the url "https://www.metronieuws.nl/in-het-nieuws/feed/" in your browser, you already see those warning messages right there. And those responses (rss feeds) are not generated for each client individually if I'm not mistaken (at least it looks like a response from AWS Cloudfront that is cached for 60 seconds). Maybe this is an issue in the feeds owners rss generator tool..?

If the owner of that feed wants to embed twitter content they should probably use something like iframes, right? Probably much easier to embed external content in your feed. And those iframes should also be able to ask the users for their constent if they have to.

As to the question of how to deal with the consent buttons.. I think I'd rather find a solution that solves it for all clients (server side) rather than having to come up with something where all clients (web / android / ios) need to handle those consent dialogs themself.. Wondering if there is something like "Ninja Cookie" that we could use on the server side to load feeds but automatically disable all cookies / consent dialogs. But I think as there is no standard API for those dialogs that this going to be quite difficult as we to handle many special cases..

Do you already know of a rss feed that uses some kind of cookie / tracking banner? I haven't seen any issues so far.

radoeka commented 3 years ago

Do you know if feedly is using the same rss api?

No I don't know.

BTW: I enabled 'full text' yesterday, and it looks like that this takes care that the twitter warning is gone. At least I've not seen it anymore.

I'll leave it enabled for some more days. If desired I can disable 'full text' and see what happens then.

In case 'full text' solves the issue, I would never ever relate this (full text) with those (twitter) warnings.

Grotax commented 3 years ago

Well the full text feature uses a library to parse the actual website I think it's basically a port of the same functionality Mozilla uses in Firefox if you use this distraction free reading.

The rss reader itself just takes the rss and parses the standard, we then remove potentially dangerous elements from the feed with http://htmlpurifier.org/

radoeka commented 3 years ago

Indeed when toggling the 'full text' functionality for the stream, the warning either appears or disappears.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.