YouTube link previews not loading.

schmeat commented 4 years ago

[X] I have searched open and closed issues for duplicates
[X] I am submitting a bug report for existing functionality that does not work as intended
[X] I have read https://github.com/signalapp/Signal-Android/wiki/Submitting-useful-bug-reports
[X] This isn't a feature request or a discussion topic

Bug description

Link preview fails to load every time a youtube video is pasted in.

Steps to reproduce

Copy a youtube link and wait for preview to fail loading.

Screenshots

https://gfycat.com/wateryunhappycommabutterfly

Device info

Time : 1570681676210 Device : samsung SM-G975W (beyond2qltecs) Android : 9 (G975WVLS2ASI6, PPR1.180610.011.G975WVLS2ASI6) ABIs : arm64-v8a, armeabi-v7a, armeabi Memory : 68M (23.08% free, 512M max) Memclass : 256 OS Host : 21HHAD01 First Version: 530 App : Signal 4.48.16 (5452)

Link to debug log

https://debuglogs.org/f132028ab42abbc37ed834e8a8b7d276d43d6f970f34db28694284c58186f69a

schmeat commented 4 years ago

Seems to have started working again. I'll assume temporary back end issues?

Edit: I spoke too soon, stopped working again.

chynric commented 4 years ago

Same for me. It is neither loading in Signal Android, nor in Signal Desktop. Signal Android: 4.48.14 Signal Desktop: 1.27.1

Flupkees commented 4 years ago

Same for me

mikeymop commented 4 years ago

I am experiencing this on 4.47.7 so I think it is in the backend.

ioistired commented 4 years ago

YouTube likely blocked Signal's crawlers. This happened to Discord and Telegram recently, as well.

Flupkees commented 4 years ago

Ah, that's good to know. I assumed all previews didn't work anymore, but I just tested and reddit previews work, so it seems to only be youtube. Does anyone know or can anyone point me to why signal is doing this in a centralised way? If I remember correctly, WhatsApp clients generate link previews locally and that seems to work really well and not only for 4 websites.

awaitlink commented 4 years ago

@Flupkees Does anyone know or can anyone point me to why signal is doing this in a centralised way?

https://signal.org/blog/i-link-therefore-i-am/

In fact it's 5 websites now, Pinterest was added by PRs from a user (who apparently works for Pinterest) as well: #8760.

Flupkees commented 4 years ago

@Flupkees Does anyone know or can anyone point me to why signal is doing this in a centralised way?

https://signal.org/blog/i-link-therefore-i-am/

In fact it's 5 websites now, Pinterest was added by PRs from a user (who apparently works for Pinterest) as well: #8760.

Thanks for that! I also found this as to why one may want to avoid whatsapp's implementation: https://techcrunch.com/2017/06/15/should-whatsapp-let-you-disable-url-previews/

thesurg3on commented 4 years ago

I also have this issue. Are there any solutions for this?

ioistired commented 4 years ago

@thesurg3on Not for us users. The Signal team has to work it out with YouTube.

schmeat commented 4 years ago

Can we please keep discussion off this ticket.

Any discussion should happen on the signal forums.

If this happens to you as well, please thumbs up the post but don't add an additional comment.

ioistired commented 4 years ago

This issue appears to be resolved.

greyson-signal commented 4 years ago

Ok, so tentatively YouTube is working again. Fingers crossed it stays that way :)

nanowinner commented 4 years ago

I'd hate to do this, but the issue is present still both on the Desktop app (latest) and Android app (latest).

chynric commented 4 years ago

Ok, so tentatively YouTube is working again. Fingers crossed it stays that way :)

Yes it was working again a short time until stopping once again.

KommX commented 4 years ago

@greyson-signal Should we reopen this issue or create a new one?

greyson-signal commented 4 years ago

Yeah, we had something that was working for a while, but it stopped again. This is a server-side issue with our proxy, but I'm ok with tracking it here. TL;DR is that we're being rate-limited by YouTube.

nutts0 commented 4 years ago

YouTube doesn't work here

wwwizzarrdry commented 4 years ago

How does duck duck go proxy their YouTube videos? Is that an option here?

JonasNord commented 4 years ago

Issue persists for me as well, omitting device info as issues are server side.

EDIT: What's the reason behind the thumbs down? Am i missing some etiquette or rule?

Yeah, we had something that was working for a while, but it stopped again. This is a server-side issue with our proxy, but I'm ok with tracking it here. TL;DR is that we're being rate-limited by YouTube.

jgspratt commented 4 years ago

Could this be re-written so that the client makes the request?

Could your proxy be altered so that it uses many IPs?

asbjornu commented 4 years ago

Why are you using a proxy to implement link previews? For privacy protection? What about those of us who value phishing protection over privacy? Having link previews that work are more important to me than my IP address being masked to the website being previewed.

The Open Graph Protocol exists to allow link previews regardless of website and should be used to implement this feature, without a hard coded list of supported domain names. If people want to restrict the domains that offer link previews, that can be configurable. The current implementation is almost completely useless to me as I never share URLs for any of the supported domains except YouTube.

Please reconsider the current implementation.

ioistired commented 4 years ago

Yes, it's for privacy protection. Doing the link previews client side amounts to leacking the contents of (some of) your messages to third parties, which Signal is obviously against.

jgspratt commented 4 years ago

Yes, it's for privacy protection. Doing the link previews client side amounts to leacking the contents of (some of) your messages to third parties, which Signal is obviously against.

In most cases (99% or better), I'm sharing a link I've already opened in my browser, so the security point is moot.

But an option is to add an option: "Generate previews locally". That makes everyone happy.

Or how about "Automatically generate previews locally if proxied preview fails"?

Or "Offer to generate previews locally if proxied preview fails"?

Ralms commented 4 years ago

@iomintz "Doing the link previews client side amounts to leaking the contents of (some of) your messages to third parties" LOL What? You know that is not true right? If you try to develop something and interact with other APIs you might be enlighten.

ioistired commented 4 years ago

@Ralms what? I have used third party APIs. How is that relevant here? Signal doesn't use any.

ioistired commented 4 years ago

Other than, I think Giphy, which it uses in a clever way.

Ralms commented 4 years ago

@Ralms what? I have used third party APIs. How is that relevant here? Signal doesn't use any.

Right, in fact it does, is using on the servers right now for this feature. So, I would love to see your arguments / proof of how using Open Graph would result on a data leak of messages content, being it is literally a Get request which can be isolated.

ioistired commented 4 years ago

Alice sends "check this out http://example.com/foo" to Bob example.com receives GET request from Signal app suppose Alice already signed in to example.com before example.com now knows that Alice sent a link to /foo on Signal

ioistired commented 4 years ago

At least the current implementation preserves Alice's anonymity

jgspratt commented 4 years ago

@iomintz, that's why a "Failed to load preview via proxy. Do you want to load it locally? This may tell example.com that you're using Signal. (click here to never show this for example.com again" message would be handy.

Ralms commented 4 years ago

Alice sends "check this out http://example.com/foo" to Bob example.com receives GET request from Signal app suppose Alice already signed in to example.com before example.com now knows that Alice sent a link to /foo on Signal

Not true, like I said, you can isolate it and for the perspective of example.com would be a generic chrome or safari browser access. And on top of that Alice was already signed in and most likely was on example.com where she obtained the link.

This level of paranoia is what will inhibit us to convince our friends to get out of things such as WhatsApp. Worst case scenario should be optimal to be enabled by whoever doesn't mind to use it.

jonest commented 4 years ago

I think there's another issue where Alice send a link to Bob. Bob has not been to www.example.com. Bob's Signal app loads a preview and now that website knows Bob.

Am I mistaken here?

ioistired commented 4 years ago

@jonest, Alice's client would generate the link preview and include it in the message sent to Bob.

jonest commented 4 years ago

@iomintz cool, thanks! Wasn't sure if that would be delivered as part of the payload or not 👍

mueller-ma commented 4 years ago

Please use the forum to discuss changes of this feature: https://community.signalusers.org/t/link-preview-when-pasting-or-typing-links/2370

asbjornu commented 4 years ago

Yes, it's for privacy protection. Doing the link previews client side amounts to leacking the contents of (some of) your messages to third parties, which Signal is obviously against.

This is blatantly false. Performing an HTTP GET request towards a URL does not expose anything but my IP address. And if the sender (as you suggest, @iomintz) is the one that actually bundles up the preview, the receiver's IP address wouldn't even be exposed, it would be the sender, who would already have had to visit the page for it to be shared in the first place.

I don't really understand what there is to protect by using a proxy here. It's the wrong solution to a non-existing problem, and exactly like @Realms writes; the level of paranoia that makes it impossible to convince our friends and family to get out of things such as WhatsApp, Facebook Messenger, etc.

greyson-signal commented 4 years ago

This is blatantly false. Performing an HTTP GET request towards a URL does not expose anything but my IP address.

Not true, like I said, you can isolate it and for the perspective of example.com would be a generic chrome or safari browser access.

Nope, this does leak some message contents. Just requesting the HTML content of a page is not what a normal web browser does. They make tons of requests for linked assets and stuff. Just requesting HTML would be easily identifiable as a request for a link preview, at which point you've leaked part of your message -- namely, the link that you're sending.

When trying to make this feature, I had even come up with ideas to do a full webpage load in an offscreen webview to make it look like legitimate web traffic... but then you need to download the image for the thumbnail, which often isn't an image a web browser would usually load, at which point you definitely look like a link preview request.

This is something where the devil is really in the details. We are looking into alternative ways to do link previews more generically, but it's tricky.

But as previously noted, people should be discussing this on the forum. We use github for bugs, not general discussion.

mikeymop commented 4 years ago

What is you opinion on removing the link preview code until a solution is found and developed?

Currently, users can see the link preview attempt to generate and then it disappears.

This can look to the user like something in Signal is broken. Then this can be revisited in a PR discussion and the issue won't continue to be bumped.

Ralms commented 4 years ago

What is you opinion on removing the link preview code until a solution is found and developed?

Currently, users can see the link preview attempt to generate and then it disappears.

This can look to the user like something in Signal is broken. Then this can be revisited in a PR discussion and the issue won't continue to be bumped.

Doing that will result on this feature most likely never being added again.

asbjornu commented 4 years ago

@greyson-signal,

Nope, this does leak some message contents.

That depends on your interpretation of "message contents". Most people will read that as "the private message the sending party writes to the receiving party", which under no circumstance will be leaked to a website being previewed, unless the preview mechanism is exceptionally broken.

Just requesting the HTML content of a page is not what a normal web browser does. They make tons of requests for linked assets and stuff.

So don't use a full-blown browser to create the preview. Just do an HTTP request with URLSession or HttpURLConnection, fetch the Open Graph Protocol metadata from the <head> section and create a preview based on that. Worst case fallback: use the <title> element to at least provide a human readable title of the link.

Just requesting HTML would be easily identifiable as a request for a link preview, at which point you've leaked part of your message -- namely, the link that you're sending.

What would be identifiable is that the user agent isn't executing JavaScript. The fact that there wasn't made a request to "linked assets and stuff" is something most sites don't correlate, as data about these requests are stored server-side on CDNs and not souped up by the JavaScript-based trackers used everywhere.

So yes, the shared site will "at best" know that a request was made from a user agent that didn't execute JavaScript. Just like any web crawler, bot, or similar, in existence.

All of these points are moot if the link is shared from Safari, though. Safari's share sheet will expose all of the shared site's metadata through a PKLinkMetadata object (I suppose something similar is possible with Intent.ACTION_SEND on Android), making a second preview request unnecessary.

This is something where the devil is really in the details.

To guarantee 100% privacy and anonymity; yes. But most (potential) users of Signal don't care about that to such an extreme extent. Some do, so please make sure they can turn link preview off, but for the rest of us who care much more about Signal being a product we in good faith can recommend to our friends and family – working and ubiquitous link preview is essential. Not having them feels like being thrown back into the early 2000's before link previews existed.

We are looking into alternative ways to do link previews more generically, but it's tricky.

PKLinkMetadata and The Open Graph Protocol are two generic ways to do this. I don't think you're using any of them.

But as previously noted, people should be discussing this on the forum.

We are, but as the Signal developers don't seem to be present there, the discussion isn't of much value, tbh.

We use github for bugs, not general discussion.

I think the title of this issue – "YouTube link previews not loading" – is a bug, and most of the comments I've read on here are discussing viable solutions to the problem, which I think is pretty much on topic.

navid-zamani commented 4 years ago

But most (potential) users of Signal don't care about that to such an extreme extent.

That is patently false, and a dangerous mindset. Such ”extreme“ (read: actually 100%, so actually having a point at all) privacy is the whole point of using Signal over e.g. just usinf WhatsApp. It is why users install it, and what they expect.

That meme of the careless ”average user“ is also very dead. Especially nowadays in the EU. Since the GDPR, even average grandparents have become very wary of their privacy not being protected. It has become fashionable.

I just had to add that, so Signal does not veer off in a suicidal direction of half-assing what its whole point is.

appdoo commented 4 years ago

First of all I think this feature should work for all links - if the user opts in.

Implementation: I understand the privacy concerns, so does anyone see a problem with the following implementation:

Sender posts a link.
If he opted in to link previews, a link preview is generated just for himself
Depending on the implementation, the message is tagged with a link preview tag.
The receiver receives the link in a message
If the Receivers opted in to „load link previews“ a link preview is generated locally for him.

This way link previews will never be transferred, they‘ll always stay local on the users device. The sender sends just a „load_link_preview“ tag. If the receiver wants to see it from youtube or any other site, he generates it‘s own preview tag ..

Information is never transferred.

Another solution is also available:

same as above
same as above
Sender sends the image to the receiver, so the receiver doesn't have to download the link preview. Even simpler solution.

Just wanted to add a note: Basically the problem can be reduced to email applications loading remote content. Personally I've opted out from loading remote content in my mail programs, but I can select "load remote content" separately for each mail.

Link previews/document preview are a great feature to know what is behind the link and helps the sender and the receiver to organise its content.

trafenrest commented 4 years ago

I prefer your second approach.

mueller-ma commented 4 years ago

As Greyson, one of the maintainers, noted above:

But as previously noted, people should be discussing this on the forum. We use github for bugs, not general discussion.

Please use https://community.signalusers.org/t/link-preview-when-pasting-or-typing-links/2370 to discuss how this feature should be improved.

sougatc commented 4 years ago

This issue is occurring again. Can someone please help debug this.

jgspratt commented 4 years ago

That's WhatsApp. Signal's text bar says "Signal message".

On Tue, Jul 21, 2020, 17:38 sougatc notifications@github.com wrote:

This issue is occurring again. Can someone please help debug this.

[image: Screenshot_20200722-030650.jpg] https://user-images.githubusercontent.com/64637913/88109611-754b7a80-cbc8-11ea-9b31-10d061888d7c.jpg

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/signalapp/Signal-Android/issues/9086#issuecomment-662119917, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI63KZC7XBBXKH6VGXZ3Z3R4YDDTANCNFSM4I7HW5VA .

sougatc commented 4 years ago

Sorry can you please elaborate more on this ? Do you mean the issue is at WhatsApp end ?

jgspratt commented 4 years ago

Your screenshot is from WhatsApp.

On Tue, Jul 21, 2020, 18:20 sougatc notifications@github.com wrote:

Sorry can you please elaborate more on this ? Do you mean the issue is at WhatsApp end ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/signalapp/Signal-Android/issues/9086#issuecomment-662136293, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI63KZLULLAEFBKAAGIQS3R4YIEFANCNFSM4I7HW5VA .

trafenrest commented 4 years ago

This issue is occurring again. Can someone please help debug this.

It was never gone

signalapp / Signal-Android