RSS feed items not downloading

[sachaw]

After adding a feed and and the appropriate rules, I get the expected number of matches and nothing ever comes of it, nothing in my rtorrent log either.

[SanPilot]

Probably the same as issue #433. I've been working on a fix (#442), but haven't been able to successfully merge yet.

[sachaw]

Yea, Here's my rss for reference http://horriblesubs.info/rss.php?res=1080

[SanPilot]

Actually, this RSS looks like it should work with Flood. Have you tried putting the magnet links in manually?

On Mon, Oct 9, 2017, 11:20 PM Sacha Weatherstone notifications@github.com wrote:

Yea, Here's my rss for reference http://horriblesubs.info/rss.php?res=1080

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jfurrow/flood/issues/461#issuecomment-335348805, or mute the thread https://github.com/notifications/unsubscribe-auth/AK822T0bL0tjoX5HAGBhkQod2AFKpPvgks5squKCgaJpZM4PyGeb .

--

Hussain Khalil hussainpilot05@gmail.com

[sachaw]

Seems to be behaving now (sort of), it never added the existing torrents in the feed, only new ones.

[sachaw]

Now, it wont download the the metadata for the magnet and is just stuck listed as a hash, I'm not having this issue with other torrent clients

[dcousens]

The RSS feed at https://www.archlinux.org/feeds/releases/ fails to work in Flood too, is this a Flood issue or? It does match and notify each item, but, the .torrent files fail to be added.

Maybe if the link fails to parse, support clicking the notification to open the URL for manual adding?

[SanPilot]

Try @jfurrow's new branch. @dcousens, this should fix your problem.

[jfurrow]

I just merged the branch, so @SanPilot it should be working for your specific use case.

Unfortunately for @dcousens it's not going to work for you, at least not with the feed you provided, for a number of reasons. Right now Flood checks for the link element first, and uses that if it exists. If not, it'll look for the enclosures element, expecting it to contain child elements with a url attribute, and downloads all of the children. The feed @dcousens provided has a link element whose content is a string that appears to be a URL of the description's release (e.g. https://www.archlinux.org/releng/releases/2017.09.01/), so Flood's going to try to download that, and it's obviously not going to work. The feed does contain the torrent URL on the enclosure (singular) element's url attribute, which Flood doesn't even look for.

It seems like tracker feeds are all over the place. Does anyone know if there's a standard? How does ruTorrent handle this? That's probably the best place to look.

One reliable solution would be to allow users to specify whatever element they want Flood to consider the torrent URL (and also what element they want to match their regex against). They'll also need to specify whether or not the torrent URL exists as the value of the element, or if it's an attribute on the element.

[dcousens]

Thanks for the in-depth response @jfurrow. It doesn't seem like you are faced with an easy problem.

[jfurrow]

The rutorrent plugin source, which @SanPilot linked, is what we should emulate, to some degree. But god damn, this logic is really hard to interpret, and there's a lot of it. I tried to summarize what it does, but I don't have time to keep interpreting it. So, what follows is only a partial interpretation. And it might not be 100% accurate.

• Iterates over every element in the feed recursively (I'm assuming it's recursive, because it doesn't look for the enclosures element — just looks for enclosure elements)
• Extracts a string from each element and stores it in variable $temp. I think this string is used to match regex rules against, but I could be wrong, because it explicitly grabs the url property on some of the elements. Weird, but that's probably the fault of inconsistent tracker feeds.
  • It handles elements enclosure, source, and title specifically. It extracts a string from the url property on enclosure or source, and extracts the child of title element, which is presumably a string (and then replaces all whitespace characters with a single space). So it's attempting to extract foo from these elements: <enclosure url="foo">, <source url="foo">, or <title>foo</title>. This is confusing to me — the url property of enclsoure and source probably contain URLs, but surely the children of <title> wouldn't be a URL, right? Okay, so maybe it's not extracting URLs... what is $this->search doing? Probably matching the regex rules against these strings, right?
  • If the element's tag isn't any of those three, then it just extracts the string content foo of the current element: <someElement>foo</someElement>
  • If the element's tag is description, then it decodes the string with html_entity_decode (presumably this is similar to JavaScript's decodeUriComponent
• While iterating, it stores all of these $temp strings it finds in an object, mapping the found strings to the element tag that contained it. In JavaScript, this object would look like this: {enclosure: 'foo', title: 'foo'}
• Depending on which keys are present in the object, a variable $href is assigned some values. Here's where the $href value is assigned, but I'm giving up for now. There's more logic after this, and I don't have time right now to keep interpreting it.

After this, the plugin handles other types of feeds (I'm discerning this from this comment), and has a ton of logic specific to this other type of feed.

I'm leaving these notes for future me, maybe. But if someone wants to implement this, I would be thrilled. Let's just try to keep the code as clean and legible is possible.

[dcousens]

I think showing a failed notification for the failed "download" would be a major improvement, not knowing either way is frustatingly hard to track.