urlpreview: Migrate away from Synapse URL Previews

coffeebank / coffee-maubot

Matrix bot plugins for Maubot 🍢 Add link previews, choose, and other tools to your Matrix chat ☕

https://coffeebank.github.io/coffee-maubot

11 stars 5 forks source link

urlpreview: Migrate away from Synapse URL Previews #8

Closed coffeebank closed 1 year ago

coffeebank commented 1 year ago

https://github.com/coffeebank/coffee-maubot/commit/7009c54f3b274c9d9c508353772bf61cea0422bd#commitcomment-125372320

@peterrus

Hey @coffeebank can you give some context around the choice to not fetch data through the Matrix API anymore? For me that was exactly what made the plugin interesting so I might want to start maintaining a fork but maybe there are some caveats I am not yet aware off.

Thanks!

coffeebank commented 1 year ago

@peterrus There were mainly three reasons for my choice to stop using the Matrix API:

Matrix Synapse API required that the server the bot account is on (1) is on Synapse and (2) has the Previews API enabled
- The docs stated "Synapse includes support for previewing URLs, which is disabled by default", which did not seem like a promising API to rely on
The API was limited in supported meta tags/fields
- v0.2 without Matrix API now supports a direct image embed, which was not possible with Matrix API
- In the future, support for more meta tags may be added
The API required an authenticated token for every request, is ratelimited, and returns inconsistent results

I have thought about maintaining a Matrix API fallback, but could not think of enough benefits.

Feel free to let me know your thoughts. I am open to bringing it back.

coffeebank commented 1 year ago

Update: In light of severe regressions for popular sites (Reddit, NY Times, Twitter, and more), urlpreview v0.3.x will be restoring Matrix API.

peterrus commented 1 year ago

Too bad it didn't work out @coffeebank. Did you try setting a user agent (a popular version of Chrome/Firefox for example?)

For me: The fact that this plugin uses the Matrix API was it's big selling point, because it does not leak IP's to the servers that the preview gets loaded from.

coffeebank commented 1 year ago

Currently, urlpreview just piggybacks off the default self.http.get from Maubot (might improve in the future™)

My thoughts have been towards Maubot instances hosted on a VPS/public server, where the IP is not a concern....

If it interests you, v0.3.x can allow you to disable htmlparser (fetching directly). I've rebuilt it to be more modular.

However, v0.1.x should be FOSS and stable, and I hope it serves your needs well :')