URL preview shows error on JetBrains YouTrack links

[babolivier]

Seen just now in #synapse-dev:

The link is https://youtrack.jetbrains.com/issue/PY-54833/Pycharm-deletes-existing-poetry-environment

While Synapse itself is not failing to generate the preview, it seems to not be sending the correct data to the remote end. I've tried it in a few other messengers. Discord seems to also fail to generate a preview but in this case just doesn't show anything client-side:

Slack on the other hand seems to be succeeding:

[DMRobertson]

The source for that page

```html

```

There isn't any useful metadata in the head that I can see, nor any fallback text in the body.

it seems to not be sending the correct data to the remote end.

What makes you say this? I would guess that YouTrack serves up different HTML if the user agent says you're Slack.

I've tried it in a few other messengers. Discord seems to also fail to generate a preview but in this case just doesn't show anything client-side:

[babolivier]

What makes you say this?

Well it obviously makes the decision about the data it sends back based on the info we send it.

To be clear I'm not necessarily saying that we should eg impersonate Slack. The resolution to this issue could very well be being better at detecting this kind of errors and just refusing to generate a preview in such cases (which Discord seems to do).

[clokep]

I investigated this a bit more (out of curiosity) and as @DMRobertson pointed out the page that gets returned is rather useless if you don't have JavaScript enabled. (It pretty much is a page saying "enable JavaScript" and then rebuilds the entire page via AJAX.)

Other things that I considered while investigating:

• They do not have oEmbed autodiscovery in the HTML (or as a response header, although Synapse doesn't support that today).
• The favicons given in the HTML are broken (they're missing an initial slash so are treated as relative instead of static), but it seems that browser default back to /favicon.ico in this case.
• The message being rendered has a hidden attribute on the div, maybe we should ignore those when generating the description from the HTML?

---

Well it obviously makes the decision about the data it sends back based on the info we send it.

Yes, but it is possible that Slack has custom code to hit an API endpoint or do something different here, or maybe they're rendering with JavaScript enabled somehow?