resolving magnet links with multiple videos

[pikdum]

Hey, just wondering if there's support for resolving a magnet link that contains multiple video files. When I try, it seems to just be returning a single video.

Currently testing with only Real-Debrid configured. Just getting into Kodi addon development, so still getting used to this.

Thanks!

[Gujal00]

ResolveURL by nature is to resolve a embed url to a playable url and is meant to be used for a single file. It is not designed to return multiple links or other attachments such as subtitles If you are linking to sites like nyaa who carry season packs in a single magnet, resolveurl is not designed for that purpose. Have a look at Kaito that scrapes nyaa and plays season packs.

[pikdum]

Right, makes sense. My use case is exactly that, trying to get episodes out of season packs since some of them are no longer seeded individually.

[pikdum]

I was able to get a poc working with basically one change:

• add optional selected_file parameter to get_media_url

That way when resolving a magnet batch you can specify which file you want to grab out of it, rather than always defaulting to the largest. Downside is you need to know what's in the magnet beforehand, but that list can usually be scraped from wherever it came from.

If this is a feature you're interested in adding, any thoughts on the implementation at https://github.com/pikdum/ResolveURL/commit/7797457df84b1da4c6df2ff49722fc2b1bbdbca3? I've only added it to the Real-Debrid plugin, since that's all I have.

Some things I haven't looked into yet:

• if this accidentally breaks the other resolvers
• level of work to add to the other resolvers

[Gujal00]

Yes your idea is sound, but as you mention you need to know the exact file name inside the magnet beforehand. Instead, I was thinking of an option to return a file list with a new optional file_list parameter. That way the addon doesnt need to scrape the magnet beforehand and it can select the file from the returned list. What do you reckon? Yes it needs support to be added on all resolvers that support magnets. There are only five currently, I have accounts with Debrid-Link, Alldebrid and Premiumize. @Twilight0 has Linksnappy account and you have Realdebrid account and between us we can add the batch support once we agree a way forward

[pikdum]

Yeah, that definitely sounds like it'd be a more convenient interface.

A somewhat related thing I noticed is that cache hits are much better when selecting all files from a batch rather than just the one you want: https://github.com/pikdum/ResolveURL/pull/1/commits/c563d4a8832391c8cb712346e106c8f94f05e036 Not sure if this behavior is specific to Real-Debrid, though.

[host505]

@pikdum thanks for bringing this feature to the table. @Gujal00 you mean something like this? https://github.com/host505/ResolveURL/commit/c020a90d27d803c4f5d6d4339f9bb8051ba305ea edit: I tried this with my add-on and it worked as expected

[pikdum]

That's pretty cool. Might even work as a default behavior when there are multiple video files, rather than choosing the largest one.

I think there's value in being able to specify the filename you want too, though, if it's known beforehand. That way it could be integrated more deeply with the video plugin.

It's sort of what I'm trying to do with https://github.com/pikdum/plugin.video.haru and https://github.com/pikdum/ResolveURL/tree/improved-batch-support.

[Twilight0]

Yes your idea is sound, but as you mention you need to know the exact file name inside the magnet beforehand. Instead, I was thinking of an option to return a file list with a new optional file_list parameter. That way the addon doesnt need to scrape the magnet beforehand and it can select the file from the returned list. What do you reckon? Yes it needs support to be added on all resolvers that support magnets. There are only five currently, I have accounts with Debrid-Link, Alldebrid and Premiumize. @Twilight0 has Linksnappy account and you have Realdebrid account and between us we can add the batch support once we agree a way forward

I don't have a permanent account on linksnappy, I only request a trial when there is a case for plugin update.

[Gujal00]

@pikdum thanks for bringing this feature to the table. @Gujal00 you mean something like this? host505@c020a90 edit: I tried this with my add-on and it worked as expected

I didnt mean the user should select the file. what I meant was SMR returning the list of filenames and urls from the magnet back to addon and then the addon regex the filenames in the list to match the season/episode that is being requested, picks the right url and plays without user intervention, like it does in seren for e.g.,

[Gujal00]

That's pretty cool. Might even work as a default behavior when there are multiple video files, rather than choosing the largest one.

Default behaviour of returning largest file is correct for movies and shouldn't change, otherwise users will see dialogs when not required.

[pikdum]

Thoughts on an interface like this?

import resolveurl

magnet = "example"

# unchanged existing behavior
resolved_url = resolveurl.HostedMediaFile(url=magnet).resolve()
# "https://example/largest.mkv"

# new behavior
resolved_urls = resolveurl.HostedMediaFile(url=magnet, resolve_all=True).resolve()
# [
#   {
#     "file": "largest.mkv",
#     "url": "https://example/largest.mkv"
#   },
#   {
#     "file": "example.mkv",
#     "url": "https://example/example.mkv"
#   },
#   {
#     "file": "foo/bar.mkv",
#     "url": "https://example/foo/bar.mkv"
#   }
# ]

Some benefits to resolving all files and then filtering out what you want afterwards, at least with Real-Debrid:

• less cache fragmentation, since "batch with all files selected" and "batch with episode 1 selected" are cached separately
• more cache hits, due to the above and since "batch with all files selected" is the default behavior in web interface
• seems really straightforward to implement

[Gujal00]

Correct, that was exactly my thought, returning a list of dicts containing filename and link.

[
 {
  'name': '[Aeenald] Karakai Jouzu no Takagi-san S01 - 03 (BD 1080p HEVC-10bit).mkv',
  'link': 'https://seedxx.debrid.link/dl/vqEzd-5TQnwczhf4K61/8986200axxxxxx90824/1/%5BAeenald%5D+Karakai+Jouzu+no+Takagi-san+S01+-+03+%28BD+1080p+HEVC-10bit%29.mkv'
 },
 {
  'name': '[Aeenald] Karakai Jouzu no Takagi-san S01 - 02 (BD 1080p HEVC-10bit).mkv',
  'link': 'downloadUrl=https://seedxx.debrid.link/dl/vqEzd-5TQnwczhf4K61/8986200axxxxxx90824/2/%5BAeenald%5D+Karakai+Jouzu+no+Takagi-san+S01+-+02+%28BD+1080p+HEVC-10bit%29.mkv'
 },
 {
  'name': '[Aeenald] Karakai Jouzu no Takagi-san S01 - 01 (BD 1080p HEVC-10bit).mkv',
  'link': 'https://seedxx.debrid.link/dl/vqEzd-5TQnwczhf4K61/8986200axxxxxx90824/3/%5BAeenald%5D+Karakai+Jouzu+no+Takagi-san+S01+-+01+%28BD+1080p+HEVC-10bit%29.mkv'
 },
 {
  'name': '[Aeenald] Karakai Jouzu no Takagi-san S01 - 04 (BD 1080p HEVC-10bit).mkv',
  'link': 'https://seedxx.debrid.link/dl/vqEzd-5TQnwczhf4K61/8986200axxxxxx90824/4/%5BAeenald%5D+Karakai+Jouzu+no+Takagi-san+S01+-+04+%28BD+1080p+HEVC-10bit%29.mkv'
 }
]

However in case of Alldebrid, the links would be uptobox links, so you need to resolve again in code. the below code would handle that as an example

links = resolveurl.resolve(magnet, resolve_all=True)
url = ''
for link in links:
    m = re.search('s{0}\s*[-e]\s*{1}'.format(season, episode), link.get('name'), re.I)
    if m:
        url = link.get('link')
        break
if url:
    if resolveurl.HostedMediaFile(url):
        url= resolveurl.resolve(url)

If that looks usable, then yes we can start coding resolveurl for the functionality

[pikdum]

Yeah, that looks good to me.

[pikdum]

Made a draft PR; let me know what you think.

[Gujal00]

I think it may be a RD issue, but looks like you are making a POST for each link url, and for season packs, could be anywhere upto 52 calls (e.g., Gintama) which could lead to flooding and RD temporarily banning your ip for 30 minutes or so. Need to be careful here

[pikdum]

Yeah, there doesn't seem to be any other way to do it though. Just tested their web interface, and it does the same thing.

Tested with Legend of the Galactic Heroes, 110 episodes, and didn't have any issues, so hopefully it's fine.

[Gujal00]

I think we should make it a two stage process if possible as I mentioned for Alldebrid (links are uptobox likns which need to be resolved again) that is in case of RD, the resolve_all returns names and http://realdebrid.smr/{file_ids}, the addon then scrapes the names for the correct season/episode match and then passes the file id back for resolution and gets the actual stream url. This will reduce the numbers of apicalls. We can write a plugin to handle http://realdebrid.smr/{file_ids} What do you reckon? If you think making 120 calls per 20 minute episode wont get your IP banned, then fine we can merge it the way you have there. Excessive apicalls have forced RD to revoke ResolveURL client-id before and I dont want that to happen again

[pikdum]

I'd rather not make so many API calls, even if there aren't any issues. Would need to think up a new interface though, if we're going to separate it into two steps or similar:

1. get all files
2. unrestrict the ones you choose

[pikdum]

Or actually, just pushed a fix that might work: https://github.com/Gujal00/ResolveURL/pull/8/commits/cd20cdc1c5eb32a6b690c236342ff6e9b19f1186 Since you mentioned AllDebrid would need to be resolved again anyways, the interface remains unchanged:

    resolved_urls = resolveurl.resolve(magnet, resolve_all=True)
    resolved_url = next(filter(lambda x: x["name"] == selected_file, resolved_urls))[
        "link"
    ]
    if resolveurl.HostedMediaFile(resolved_url):
        resolved_url = resolveurl.resolve(resolved_url)
    play_item = xbmcgui.ListItem(path=resolved_url)

[Gujal00]

If you have time can you see what is required to add this feature request https://github.com/jsergio123/script.module.resolveurl/issues/439 Basically we need to add a function which gets a list of magnet ids and return a list of magnet ids that are cached ready for instant play This will improve functionality in any addon that scrapes torrent sites which can then only look for cached torrents as an option

[pikdum]

It could be added now pretty easily, but it'd be pretty inefficient with API calls. To not have issues, we'd need a __check_cache_multi or similar function added to every plugin that:

• takes in a list of magnets to check
• uses the correct provider API to check multiple
• formats output in something easy to parse, like [{'magnet': '', 'cached': True}]

(Or refactor the existing __check_cache to support this)

After that, it'd be pretty straightforward to wire up everything. Maybe something roughly like:

# could add to hmf.py or similar
def find_cached(magnets):
    cached = []
    # combine results from configured resolvers
    for resolver in resolvers:
        cached = list(set(cached + resolver.find_cached(magnets)))
    return cached

# add to each provider plugin
def find_cached(self, magnets):
    cached = []
    # [{'magnet': '', 'cached': True}]
    for magnet in self.__check_cache_multi(magnets):
        if magnet['cached']:
            cached.append(magnet['magnet'])
    return cached

Might be useful if you wanted to add some visual distinction of what can be played instantly and what might take a few minutes, I guess.

[Gujal00]

Most debrid providers accept a list of magnet hashes and return their cache status in a single api call, so should be fine there. the existing __check_cache function in each plugin can be enhanced to handle this by looking at the input if it is a string or list and do accordingly

The visual distinction would be upto the actual addon. Resolveurl is only providing an utility function to get the cache status of a list of magnets.

[pikdum]

The visual distinction would be upto the actual addon. Resolveurl is only providing an utility function to get the cache status of a list of magnets.

Right, was just brainstorming a potential use of this feature. I'm not sure I'd personally bother wiring something up with this in my addon though, since it's only a minute or so wait usually if uncached.

Could see that feature being useful if building a more complicated source select system, though, like what Seren and others have. But my addon is super basic in comparison. :)

[host505]

@Gujal00 I made a new method to check a list of hashes. Too many if/elses to toss it into the __check_cache one imo. Also, __check_cache won't work outside of smr because of the double underscores (I came to know the hard way). The idea as you said is: add-on sends a list of magnet hashes to smr, then smr returns a list of those that are cached.

    def check_cache_list(self, hashes):
        if isinstance(hashes, list):
            _hash = '/'.join(hashes)
            try:
                url = '%s/%s/%s' % (rest_base_url, check_cache_path, _hash)
                result = self.net.http_GET(url, headers=self.headers).content
                js_result = json.loads(result)
                cached_hashes = []
                for h in hashes:
                    if h in js_result and len(js_result.get(h, {}).get('rd')) > 0:
                        cached_hashes.append(h)
                return cached_hashes
            except Exception as e:
                common.logger.log_warning("Real-Debrid Error: CHECK CACHE LIST | %s" % e)
                raise

        return []

The call from an add-on would be something like

from resolveurl.plugins import realdebrid
cached_hash_list = realdebrid.RealDebridResolver().check_cache_list(unchecked_hash_list)

Edit: still needs some work/adjustments: 401 handling/refresh token etc

[Gujal00]

@host505 yes, that was the idea I had. you already have the essential logic there. The call from an addon should not be to a specific debrid provider, it should be to SMR hmf so that SMR can handle it with whichever debrid provider is available. This will need a new function in hmf Busy week at work this week, will get some time this weekend to look at how best to generalise it.

[Gujal00]

@host505 I have pushed resolve_all support to premiumize and alldebrid. Can you please test, I presume you have premiumize account

[host505]

Unfortunately I don't have a premium one anymore...

[Gujal00]

@host505 Okay no problem, I tested AD, DL, and PM with SMR link tester and it works correctly. If you add a magnet link with $$all at the end, to Link tester, it will show the playable files in the magnet and then you can manually select without the $$all at the end, it sticks to the original behaviour of playing the largest file in the magnet SMR v5.1.61 has been released with the return_all support

[host505]

I noticed, thanks. I've been testing this feature with r-d with my add-on the past 2 weeks - apart for some issues which have been reported and resolved it seems to be working flawlessly. I've now released a version which takes advantage of this feature and fetches some pack sources, so I guess if there are any issues they will be revealed sooner or later.

[Gujal00]

@host505 excellent news. hopefully people will start using and report if any issues.