Url preview endpoint

[adnull]

This PR introduces the missing URL Preview functionality ( Issue #1303 )

According to the specification that endpoint requests require authentication and are rate limited. There are both endpoints (v1 and v3) as some clients still use deprecated endpoints for url preview requests.

Key features:

• takes info from html pages or image links. Max image size downloadable is configurable. Timeout is configurable
• stores the url's image as a thumbnail in media storage if the thumbnail size for url previews is configured. Otherwise stores the original image (to make it sy-test compatible).
• uses the maxThumbnailGenerators variable to prevent overload. Waits 30s for free generator.
• keeps an in-memory cache of url preview responses. The cache entry expiration time is configurable
• stores the the url preview json responses in the media storage to prevent any extra load
• url preview blacklist is configurable as a list of regexps in the config

Missing:

• tests
• synapse's ability to query/extract oEmbed resources

I'd like somebody to take a look at the code.

• [x] I have added Go unit tests or Complement integration tests for this PR or I have justified why this PR doesn't need tests
• [x] Pull request includes a sign off below using a legally identifiable name or I have already signed off privately

Signed-off-by: Aleksandr Dubovikov <d.lexand@gmail.com>

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 68.23529% with 135 lines in your changes missing coverage. Please review.

Project coverage is 67.74%. Comparing base (6cd1285) to head (f58da42).

Files with missing lines	Patch %	Lines
mediaapi/routing/url_preview.go	68.46%	74 Missing and 49 partials :warning:
mediaapi/thumbnailer/thumbnailer_nfnt.go	54.16%	7 Missing and 4 partials :warning:
mediaapi/routing/download.go	0.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #3437 +/- ## ========================================== - Coverage 67.74% 67.74% -0.01% ========================================== Files 518 519 +1 Lines 47457 47881 +424 ========================================== + Hits 32150 32437 +287 - Misses 11311 11393 +82 - Partials 3996 4051 +55 ``` | [Flag](https://app.codecov.io/gh/matrix-org/dendrite/pull/3437/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=matrix-org) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/matrix-org/dendrite/pull/3437/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=matrix-org) | `53.06% <68.23%> (+0.21%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=matrix-org#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[S7evinK]

Thanks for contributing to Dendrite!

The repo has been cloned into element-hq/dendrite and made the switch to AGPL as per https://matrix.org/blog/2023/11/06/future-of-synapse-dendrite/. For more information, feel free to read this and this

If you still wish to contribute under the new license, feel free to create a new PR over at element-hq/dendrite.