search

hydrusnetwork / hydrus

A personal booru-style media tagger that can import files and tags from your hard drive and popular websites. Content can be shared with other users via user-run servers.
http://hydrusnetwork.github.io/hydrus/
Other
2.4k stars 158 forks source link

URL converter not honoured for File urls #697

Open bbappserver opened 4 years ago

bbappserver commented 4 years ago

Nitter has been coughing up blood lately, throwing 504s (because its app backend is responding to nginx too slowly). Nitter seems to want to serve all media by downloading and then forwarding it. To see if I could mitigate the bandwidth problem I was trying to see if I could alter the nitter parsers to use twitter as CDN for media, so that nitter only needs to worry about serving a structure hydrus knows how to parse, not any of the corresponding media files.

Simple enough I thought, I'll just change the nitter file url class to convert to the twitter CDN, make sure there is a twitter file url class installed and it should just work right?

Not so it seems. Here is my attempt

nitter file converrt

Nitter media

[50, "nitter tweet media", 9, ["4201c88c64d355d91282f83a59a6e85babf6214b4a616f69e2816b0442c1d231", 2, "https", "nitter.net", [false, false, false, true, true, false], [[[51, 1, [0, "pic", null, null, "pic"]], null], [[51, 1, [2, "media%2F.*", null, null, "media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]], null]], [], [55, 1, [[[5, "url percent encoding"], [9, ["https://nitter.net/pic", "https://pbs.twimg.com"]]], "https://nitter.net/pic/media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]], 0, [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], null, null, 1, "https://nitter.net/pic/media%2FEZv1zeRVcAEtBnz.jpg%3Fname%3Dorig"]]

Twitter media

[50, "twitter media", 9, ["efc50e7beb9aa006b00a35c2dc0a5c9eb3660c6b2d392f2382d761e39e6d700e", 2, "https", "pbs.twimg.com", [false, false, true, false, true, false], [[[51, 1, [0, "media", null, null, "media"]], null], [[51, 1, [3, "", null, null, "example string"]], null]], [], [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], 0, [55, 1, [[], "https://hostname.com/post/page.php?id=123456&s=view"]], null, null, 1, "https://pbs.twimg.com/media/ElEN80eUcAAShil?format=jpg&name=orig"]]

But when I put a twitter tweet url into a test url download page, it still tried to serve the media through nitter.

I have a strong suspicion that nitter is dying because some or several hydrus users are asking for way too much, so I really hopped that would work and we could just roll that change into the defaults a future version.

Environment

Hydrus version:414 Platform: Mac 10.14 OS specifics:

bbappserver commented 2 years ago

BUMP: Resolved for Twitter by converting nitter posts back into Twitter, still seems to be a problem for file URLS.