Prowlarr indexer support for mediafusion

vijayvcm commented 3 months ago

Issue

Prowlarr indexer support for Tamil torrents
216

Changes

Added prowlarr indexer for media fusion

NOTES

There are a few things that need improvement here

Stripping down the media fusion base URL and manifest.json to frame the URL. I couldn't move it to some variable or anything here. It's a little difficult to read.
Currently, it supports only Direct Torrents. Not sure it will work if it is configured with any debrid services. Maybe we can improve it from here.
In some rare cases the regex might not be applying properly for titles.

Summary by CodeRabbit

New Features
- Introduced a new indexer configuration for "MediaFusion - ElfHosted" that supports direct torrent searches for movies and TV shows.
- Enabled structured retrieval of media content with specific query parameters for enhanced search functionality.
- Implemented dynamic API request paths based on user input, improving the accuracy of search results.
- Enhanced user experience with metadata validation and informative responses for no-result scenarios.
- Added support for the "Prowlarr Indexer" within MediaFusion, facilitating integration with Radarr and Sonarr for improved media management.

mhdzumair commented 3 months ago

Nice work. Can you make some changes

add the indexer file into /resources/yaml/
make it generic indexer template and ask to input the addon configured manifest url. So that they can provide either from elfhosted or from their own selfhosted url.
most of the sources scraped from tamilmv and tamilblaster might be added into db with mf* id in case of not able to match with imdb id. So it would be good to support direct title search. But for title search you want to run two queries.
- first /{secret_str}/catalog/{catalog_type}/{catalog_id}/search={search_query}.json search
- then fetch the torrents with stream endpoint.

For information on our supported api endpoint you can check the swagger ui https://mediafusion.elfhosted.com/docs

Add the docs about prowlarr indexer file link in Readme and in the home.html

vijayvcm commented 3 months ago

[x] Moved the yaml file to the resources folder
[ ] I am not sure prowlarr allows to input the custom base URL. When I checked the prowlarr cardigan schema, it showed that "links" is a required parameter. So I think anybody who wants to use the custom domain has to replace it manually.
[ ] I could not find any example that uses two queries to get the streams. Probably, we can have an endpoint that combines both search and streams logic which will reduce the complexity in the yaml file.

ragmehos commented 3 months ago

Currently, it supports only Direct Torrents. Not sure it will work if it is configured with any debrid services. Maybe we can improve it from here.

Looks like the torrentio prowlarr scraper is able to take in the debrid key https://github.com/dreulavelle/Prowlarr-Indexers/blob/main/Custom/torrentio.yml#L36

mhdzumair commented 3 months ago

Even though its try to setup debrid key, its not used as it is. only prowlarr support infohash and magnet link isnt it?
i could see in that template its only setting info hash. https://github.com/dreulavelle/Prowlarr-Indexers/blob/a1e7568ec97ee8e10a4fc05240d3a038b058a854/Custom/torrentio.yml#L109

if the prowlarr support setting direct medialink then its fine to use debrid configured url mediafusion. otherwise you can just use the configured url without that.

Apart from this, you can generate the magnet link as below https://github.com/mhdzumair/MediaFusion/blob/0809a0155ecad654909dd0f8a5f54985eeb32258/kodi/plugin.video.mediafusion/lib/router.py#L267-L269 https://github.com/mhdzumair/MediaFusion/blob/0809a0155ecad654909dd0f8a5f54985eeb32258/kodi/plugin.video.mediafusion/lib/utils.py#L97-L105

mhdzumair commented 3 months ago

I could not find any example that uses two queries to get the streams. Probably, we can have an endpoint that combines both search and streams logic which will reduce the complexity in the yaml file.

I can easily set up an endpoint, but I'm afraid this will make huge impact on elfhosted infrastructure even though they have ratelimits.

Today mediafusion elfhosted faced downtime due to high number of internal scraping. They have only rate limit for public access. They recently set up an indexer for their another application name "riven" its internally scraping through mediafusion api. Therefore, until riven figure out slow scraping rather than aggressively. We enabled mediafusion in-built rate limit scraping.

if in-built ratelimit enabled, it will hugely reduce the bots scraping. including this prowlarr indexer. So, our current ratelimit criteria as below.

General API limit: 50 / 1 min
Catalog endpoint: 150 / 5 min
Stream endpoint: 10 / 1 hour 
(This mean you can open 10 movies/series/live tv per 1 hour. )

Streaming provider endpoint and poster endpoints are exclude from rate limitation since its a real user's usage endpoints.

mhdzumair commented 2 months ago

Hii @vijayvcm any updates? I'm in favor to create a new API endpoint for searching based on title and provide torrents detail. You might need to create API endpoint as well as crud function with mongo aggregation pipeline, check the search_meta endpoint function in api/main.py and the process_search_query function in db/crud.py

vijayvcm commented 2 months ago

Hi @mhdzumair

Sorry for the delay. Unfortunately, I could not spend much time on this. I am working as a full-time developer and I hardly get time to work on my personal projects.

Probably I will continue it in the coming weekend. Until then I would be happy if anybody could take the lead from here.

mhdzumair commented 2 months ago

Yeah, no pressure.

ragmehos commented 2 months ago

@vijayvcm @mhdzumair I tested the below prowlarr yml, seems to work

---
id: mediafusion-elfhosted
name: MediaFusion - ElfHosted
description: "MediaFusion - ElfHosted Free Public Indexer"
language: en-US
type: public
encoding: UTF-8
followredirect: false
testlinktorrent: false
requestDelay: 2
links:
  - https://mediafusion.elfhosted.com/
caps:
  categories:
    Movies: Movies
    TV: TV

  modes:
    search: [q]
    movie-search: [q, imdbid]
    tv-search: [q, imdbid, season, ep]
  allowrawsearch: false

settings:
  - name: mediafusion_note
    type: info
    label: MediaFusion Note
    default: "Please enter the mediafusion addon url below (manifest.json)"
  - name: addon_url
    type: text
    label: Addon URL
    default: ""
  - name: validate_imdb_movie
    type: text
    label: IMDB ID of the movie for Radarr validation (required)
    default: "tt0110357" # The Lion King
  - name: validate_imdb_tv
    type: text
    label: IMDB ID of the TV show for Sonarr validation (required)
    default: "tt11815682"  # Hacks S1 E1

search:
  headers:
    User-Agent:
      [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0",
      ]
  paths:
    - path: "{{ if .Query.IMDBID }}stream/movie/{{ .Query.IMDBID }}.json{{ else }}stream/movie/{{ .Config.validate_imdb_movie }}.json{{ end }}"
      method: get
      response:
        type: json
        noResultsMessage: '"streams": []'
      categories: [Movies]
    - path: "{{ if .Query.IMDBID }}stream/series/{{ .Query.IMDBID }}{{ else }}stream/series/{{ .Config.validate_imdb_tv }}{{ end }}:{{ if .Query.Season }}{{ .Query.Season }}{{ else }}1{{ end }}:{{ if .Query.Ep }}{{ .Query.Ep }}{{ else }}1{{ end }}.json"
      method: get
      response:
        type: json
        noResultsMessage: '"streams": []'
      categories: [TV]

  rows:
    selector: streams
    missingAttributeEqualsNoResults: true

  fields:
    title:
      selector: description
      filters:
        - name: split
          args: ["\n", 0]
    year:
      selector: description
      filters:
        - name: regexp
          args: "(\\b(19|20)\\d\\d\\b)"
    category_is_tv_show:
      text: "{{ .Result.title }}"
      filters:
        - name: regexp
          args: "\\b(S\\d+(?:E\\d+)?)\\b"
    category:
      text: "{{ if .Result.category_is_tv_show }}TV{{ else }}Movies{{ end }}"
    infohash:
      selector: infoHash
    size:
      selector: description
      filters:
        - name: regexp
          args: "\\b(\\d+(?:\\.\\d+)? [MG]B)\\b"
    seeders:
      selector: description
      filters:
        - name: regexp
          args: "(?<=👤 )(\\d+)"

coderabbitai[bot] commented 2 months ago

Walkthrough

The update introduces the "MediaFusion - ElfHosted" public indexer via the mediafusion.yaml configuration, enhancing media search capabilities for movies and TV shows. It incorporates structured metadata, dynamic API paths, and user input validation. Key features include detailed response parsing and robust error handling, ensuring an informative user experience when retrieving media content.

Changes

File Path	Change Summary
`resources/yaml/mediafusion.yaml`	New file created for the "MediaFusion - ElfHosted" indexer, detailing metadata, search modes, user input settings, and response handling.
`README.md`, `resources/html/home.html`	Added support for the Prowlarr indexer in both files, improving integration with Radarr and Sonarr for enhanced media management.

Poem

🐰 In fields of data, bright and new,
MediaFusion hops, with a view!
For movies and shows, it searches with grace,
Bringing treasures to every space.
With fields and paths, oh what a delight,
A joyful find, in day or night! 🌟

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

vijayvcm commented 2 months ago

@ragmehos Your yaml works fine for me!

But you have not used the addon_url, which allows us to use a self-hosted mediafusion instance and gives filtered results based on the addon configuration.

I have modified the code a little bit (earlier it had a few issues). If the user configured the addon URL then it takes the addon URL as a prefix to {instance}/{secret_str}/stream/{type}/{id}.json. Otherwise, it uses the default endpoint with the base URL which is an elfhosted instance {elfhosted}/stream/{type}/{id}.json.

@mhdzumair Can you tell me on which scenarios we need mf* ids? I can almost get the IMDB ID of any movie/series with the tmdb API by using their findExternalId API (even for Indian movies). I would suggest that we should stick with the public IDs as they don't change and will be easy to integrate/work with the other applications. Using our IDs will be difficult to maintain in the long run.

If the items with mf* id are less in count, then we can try to map those to the public IDs. Just my opinion.

mhdzumair commented 2 months ago

The new changes improved the IMDB ID detection for TamilMV and TamilBlasters. So yeah, I agree that we could stick into IMDb ID search. Could you just add a reference into README doc? ill merge this MR afterwards. furthermore if you want you can add into home page as list item https://github.com/mhdzumair/MediaFusion/blob/f4ff80d6270421e6d4809f5d9367358d8342dd7f/resources/html/home.html#L77

vijayvcm commented 2 months ago

@mhdzumair Updated the README.md and home.html

mhdzumair commented 2 months ago

@vijayvcm thanks for your contribution.

mhdzumair / MediaFusion