Occasionally but consistently, gotMetadataFromURL: http error 400: Bad Request

[bittaurus]

First, I'd like to say thank you for the work you have put into this, and it mostly works for me.

That being said, I have occasional .torrent files from torrenting.com that consistently give a torrent-add result: gotMetadataFromURL: http error 400: Bad Request and fail to add when processed. I can add them by hand without problem. I took the liberty of hexing my hashkey out of the torrent files if you want to see if you can figure out the buggery...

400Bad.zip

Here's the requested environment data...

[user@host ~]$ ruby -v ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux] [user@host ~]$ transmission-rss -v 1.0.0.beta1 [user@host ~]$ cat ~/.config/transmission-rss/config.yml

feeds:
        - url: https://www.torrenting.com/t.rss?download&u=[redacted]&72;1;3;2;11&tp=[redacted]
          regexp:
                  - ^.*XviD AC3.EVO.*$
                  - ^.*XviD AC3.ETRG.*$
          download_path: /home/user/Downloads/

        - url: https://www.torrenting.com/t.rss?download&u=[redacted]&29;34;30;74;21;35;76;72;1;3;2;11;38;47;49;75;27;93;26;55;73;82;18;4;5;99;88;40&tp=[redacted]&free=on
          download_path: /home/user/Downloads/

update_interval: 300

add_paused: false

server:
  host: localhost
  port: 9091
  rpc_path: /transmission/rpc

login:
  username: [redacted]
  password: [redacted]

seen_file: /home/user/.config/transmission-rss/seen

client:
  timeout: 30

log:
  target: /home/user/.log/transmission-rss.log
  level: debug

2018-08-04 08:34:05 (debug) on_new_item event http://rss.torrenting.me/download.php/1197362/The House Of Tomorrow.2018.BDRip.XviD.AC3-EVO.torrent?torrent_pass=[redacted]
2018-08-04 08:34:05 (debug) request localhost:9091
2018-08-04 08:34:05 (debug) got session id Mr2F5MMMsyKYPabxHm03wRKXVe0ugIad5c67Z5G5GzI5S1J3
2018-08-04 08:34:05 (debug) request localhost:9091
2018-08-04 08:34:05 (debug) torrent-add result: gotMetadataFromURL: http error 400: Bad Request
...
2018-08-04 08:35:46 (debug) on_new_item event http://rss.torrenting.me/download.php/[redacted]/Hogans Heroes S01-06 br hevc-d3g (2).torrent?torrent_pass=[redacted]
2018-08-04 08:35:46 (debug) request localhost:9091
2018-08-04 08:35:46 (debug) got session id Mr2F5MMMsyKYPabxHm03wRKXVe0ugIad5c67Z5G5GzI5S1J3
2018-08-04 08:35:46 (debug) request localhost:9091
2018-08-04 08:35:46 (debug) torrent-add result: gotMetadataFromURL: http error 400: Bad Request
...
2018-08-04 08:35:07 (debug) on_new_item event http://rss.torrenting.me/download.php/1181902/Saving Private Ryan 1998 4
k dual Hevc-d3g (remux) (2).torrent?torrent_pass=[redacted]
2018-08-04 08:35:07 (debug) request localhost:9091
2018-08-04 08:35:07 (debug) got session id Mr2F5MMMsyKYPabxHm03wRKXVe0ugIad5c67Z5G5GzI5S1J3
2018-08-04 08:35:07 (debug) request localhost:9091
2018-08-04 08:35:07 (debug) torrent-add result: gotMetadataFromURL: http error 400: Bad Request

[peskdale]

I have just discovered a similar issues with some of my downloads. I've done some digging, and in my case, it's % symbols in my urls that are causing issues for transmission. I imagine in the OP's case it's the spaces in the url.

Adding a good helping of URI.encode(file) to this line fixed the issue for me.

[nning]

Thank you very much for your report and the fix! I will include it later this day.

Thanks again and please excuse my late response.

[nning]

It was not as straight forward, as I initially thought. Technically, the URLs are not correct as they contain special characters that are not encoded properly.

I included a check on torrent add whether an URL is encoded. When it is not, transmission-rss encodes it now before adding.

Thanks again for your input!

[peskdale]

@nning Turns out the percentage signs weren't the issue for me, and your additional check doesn't work with my URLs.

The RSS feed I'm using is (for some insane reason) using the unicode Greek Letter Iota, instead of a capital I, in an otherwise correctly encoded url.

Your decode check outputs correctly decodes the rest of the URL, hence my weird ones are slipping through the net. I can only get these URLs to work by forcing the encode.

I'll probably just patch this in my own fork, but I'd like to understand what (if any) negative results you were able to see when force encoding all URLs?

[killakam3084]

Circling on this one. Seems a reasonable solution would be to tokenize the URI and only encode the the PATH portion of the URI potentially concatenating back w/ the original. Likely protocol/host/port and query param portions are properly encoded (probably too big an assumption)...

If only private trackers had more sanitization schemes in place for uploads sigh

[nning]

Force encoding all URLs breaks feeds with correctly encoded URLs, because there is no way for Ruby's URI.encode to determine, if Strings are already encoded. For example, % encodes to %25 and if URI.encode is called again, the percentage sign is encoded again as %25 leading to the String %2525.

As far as I understand, encoding just the path portions of the URLs does not solve any problem.

I wonder if it is more likely to be an issue with UTF8 interpretation of the input RSS feed.

[killakam3084]

Right I see that...I guess specifically what I'm trying to solve is URLs on trackers that erroneously list entries like this:

<item>
    <title>
        The Handmaid's Tale S02 BluRay 1080p DD5 1 H265-d3g
    </title>
    <link>
        https://iptorrents.com/download.php/3118982/The Handmaids Tale S02 br hevc-d3g.torrent? 
        torrent_pass=[redacted]
    </link>
    <pubDate>Mon, 03 Dec 2018 00:03:54 +0000</pubDate>
    <description>15.3 GB; TV/Packs</description>
</item>

This encodes to:

https://iptorrents.com/download.php/3118982/TheHandmaidsTaleS02brhevc-d3g.torrent

Which is of course incorrect.

Encoding just this portion: The Handmaids Tale S02 br hevc-d3g.torrent Correctly produces the valid: The%20Handmaids%20Tale%20S02%20br%20hevc-d3g.torrent

So in actuality only manipulating the portion of the path that names the torrent more or less. This is usually the portion where trackers don't properly encode. As you mention though...other issues could still persist w/ the character set.

[nning]

Are you sure, the current master encodes the URL as you said in your example? I would presume, your example is exactly the case, i tested against.

(See https://github.com/nning/transmission-rss/blob/master/spec/client_spec.rb#L62)

[killakam3084]

Haven't started debugging...but I think the issue may be this... Try to be of more help when I pull down the code.

I'm assuming with this line here file is just the filename portion of the URL?

when :url
          file = URI.encode(file) if URI.decode(file) == file

Here's output from the add_torrent procedure:

CGroup: /system.slice/transmission-rss.service └─9415 /usr/bin/docker run -v /home/iillmaticc/.config/transmission-rss/transmission-rss.conf:/etc/transmission-rss.conf --name transmission-rss nning2/transmission-rss:latest

Dec 18 22:09:08 localhost.localdomain docker[9415]: 2018-12-19 04:09:08 (debug) no privilege dropping Dec 18 22:09:08 localhost.localdomain docker[9415]: 2018-12-19 04:09:08 (debug) 0 uris from seenfile Dec 18 22:09:08 localhost.localdomain docker[9415]: 2018-12-19 04:09:08 (debug) pid 7 Dec 18 22:09:08 localhost.localdomain docker[9415]: 2018-12-19 04:09:08 (debug) aggregator start Dec 18 22:09:08 localhost.localdomain docker[9415]: 2018-12-19 04:09:08 (debug) aggregate https://iptorrents.com/t.rss?u=1779890;tp=0983d697a7641e83bff4c3bda6a218fe;bookmarks;download Dec 18 22:09:09 localhost.localdomain docker[9415]: 2018-12-19 04:09:09 (debug) on_new_item event https://iptorrents.com/download.php/2730807/The Handmaids Tale S01 br hevc-d3g.torrent?torrent_pass=[redacted] Dec 18 22:09:09 localhost.localdomain docker[9415]: 2018-12-19 04:09:09 (debug) request 172.17.0.3:9091 Dec 18 22:09:09 localhost.localdomain docker[9415]: 2018-12-19 04:09:09 (debug) got session id DVl9NtunAd0B0VNIWb5rz6zwcI4wFp4HLkgHGe0y65E08QUX Dec 18 22:09:09 localhost.localdomain docker[9415]: 2018-12-19 04:09:09 (debug) request 172.17.0.3:9091 Dec 18 22:09:09 localhost.localdomain docker[9415]: 2018-12-19 04:09:09 (debug) torrent-add result: gotMetadataFromURL: http error 400: Bad Request