Closed kiskey closed 2 months ago
please test the new improved scraping, if you're using the docker-compose then update the compose file to use "beta" image for mediafusion and dramatiq worker
Hi @mhdzumair ,
I just tried the beta image,but
The beta image seems to be crashing with module not found error "Ratelimit" .
ModuleNotFoundError: No module named 'ratelimit'
[2024-09-16 03:36:59,067] [PID 1] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '1.17.0' is booting up.
[2024-09-16 03:36:58,064] [PID 8] [MainThread] [dramatiq.WorkerProcess(0)] [ERROR] Failed to import module.
Traceback (most recent call last):
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/dramatiq/cli.py", line 393, in worker_process
module, broker = import_broker(args.broker)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/dramatiq/cli.py", line 123, in import_broker
module, broker_or_callable = import_object(value)
^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/dramatiq/cli.py", line 112, in import_object
module = importlib.import_module(modname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "
File "
File "
File "
File "
File "
File "/mediafusion/api/task.py", line 10, in
from scrapers import tv, imdb_data, trackers, helpers, prowlarr, prowlarr_feed
File "/mediafusion/scrapers/tv.py", line 10, in
from db import schemas, crud
File "/mediafusion/db/crud.py", line 30, in
from scrapers.utils import run_scrapers
File "/mediafusion/scrapers/utils.py", line 6, in
from scrapers.prowlarr import ProwlarrScraper
File "/mediafusion/scrapers/prowlarr.py", line 20, in
from scrapers.base_scraper import BaseScraper
File "/mediafusion/scrapers/base_scraper.py", line 8, in
from ratelimit import limits, sleep_and_retry
ModuleNotFoundError: No module named 'ratelimit'
Thanks, MJ
pushed the new image, make sure to re-pull the beta image or use full tag as mhdzumair/mediafusion:beta@sha256:e5fa4328c1bd516edb6bc6ddf464d4b23f61c9b4f35046f12c9e5f0d1a9e5b3e
hi @mhdzumair ,
Thanks for fixing the libraries. It works fine now for the series and movies.
But i see the catalog got some updates.So when setting up the new catalog ,
i get "An error occured while encrypting user data "
Simultaneously on the logs, i see this "encrypt-user-data" 422
INFO::16-Sep-24 06:21:20::middleware.py::49 - 192.168.1.130 - "GET http://1.mjlan.duckdns.org/static/css/styles.css HTTP/1.1" 200
INFO::16-Sep-24 06:28:43::middleware.py::49 - 192.168.1.130 - "POST http://1.mjlan.duckdns.org/encrypt-user-data HTTP/1.1" 422
INFO::16-Sep-24 06:28:50::middleware.py::49 - 192.168.1.130 - "POST http://1.mjlan.duckdns.org/encrypt-user-data HTTP/1.1" 422
INFO::16-Sep-24 06:28:53::middleware.py::49 - 192.168.1.130 - "POST http://1.mjlan.duckdns.org/encrypt-user-data HTTP/1.1" 422
INFO::16-Sep-24 06:28:55::middleware.py::49 - 192.168.1.130 - "POST http://1.mjlan.duckdns.org/encrypt-user-data HTTP/1.1" 422
INFO::16-Sep-24 06:30:50::middleware.py::49 - 192.168.1.130 - "POST http://1.mjlan.duckdns.org/encrypt-user-data HTTP/1.1" 422
INFO::16-Sep-24 06:31:13::middleware.py::49 - 192.168.1.130 - "POST http://1.mjlan.duckdns.org/encrypt-user-data HTTP/1.1" 422
@kiskey could you open the browser network tab and check the post entry and check the response message
@mhdzumair i now started seeing this in log . First i had the old key from before which i used. Which started through the same error. Then i generated openssl random base 64 key and used it. Then i see the below errors in log .
File "/mediafusion/api/middleware.py", line 90, in dispatch
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in call
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
raise exc
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in call
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mediafusion/api/main.py", line 102, in add_cors_header
response = await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
raise exc
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
await app(scope, receive, sender)
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mediafusion/utils/wrappers.py", line 23, in wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mediafusion/api/main.py", line 583, in encrypt_user_data
encrypted_str = crypto.encrypt_user_data(user_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mediafusion/utils/crypto.py", line 22, in encrypt_user_data
cipher = AES.new(SECRET_KEY, AES.MODE_CBC, iv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/Crypto/Cipher/AES.py", line 228, in new
return _create_cipher(sys.modules[__name__], key, mode, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/Crypto/Cipher/init.py", line 79, in _create_cipher
return modes[mode](factory, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/Crypto/Cipher/_mode_cbc.py", line 274, in _create_cbc_cipher
cipher_state = factory._create_base_cipher(kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mediafusion/.local/share/virtualenvs/mediafusion-ZmKbvzsi/lib/python3.12/site-packages/Crypto/Cipher/AES.py", line 90, in _create_base_cipher
raise ValueError("Incorrect AES key length (%d bytes)" % len(key))
ValueError: Incorrect AES key length (44 bytes)
should i be generating any form of specific key type in lesser length ? Also let me check the response message and let you know
the key length should be 32 characters.
@kiskey could you open the browser network tab and check the post entry and check the response message
i see 500 Internal server error in the network response for the post request. Let me correct key size and try again and update you
hi @mhdzumair ,
You are right. The key size was the issue. After inputting 32 character lenth key for secret_key . Now i don't see that issue anymore. So it works Now.
Thanks again for guiding and helping me to figure out.
Though i am wondering why it did not accept the key i have been using before with stable version 3.9.16 .
Though i am wondering why it did not accept the key i have been using before with stable version 3.9.16 .
I didn't quite understand. You can use any characters in the key.
The recent changes affect how user data is generated. Since URLs generally have a character limit of around 2k, the system now tries to reduce the size by encoding alias names in the config, encrypting, then compressing and encoding the data. Previously, the entire config was compressed first, then encrypted and encoded, which resulted in a slightly larger size.
Though i am wondering why it did not accept the key i have been using before with stable version 3.9.16 .
I didn't quite understand. You can use any characters in the key.
The recent changes affect how user data is generated. Since URLs generally have a character limit of around 2k, the system now tries to reduce the size by encoding alias names in the config, encrypting, then compressing and encoding the data. Previously, the entire config was compressed first, then encrypted and encoded, which resulted in a slightly larger size.
Thanks for explaining it. I figured it also has something to do with my cache /session . When i tried with private tab and test the manifest url generation , it worked with the old key. So session/cache has something to do.not sure.
Thanks again and now its working fine as expected.
hi @mhdzumair ,
I further tested the search results with series. Sometimes it misses the results returned . For example, the below . Say i selected on UI True detective S4 EP2 , The result seem to be fetching all the seasons and episodes, but when mediafusion filters them, it seem to drop or not matching the title properly. (Note: The search used the imdbid , so the results returned are all for the same titles )
From the result below ,
the row number 3 should have been matched, but mediafusion did not match
as its comparing wrong titles , not sure how it came up with title "Detective" vs the actual imdb title . So the entire results seems to be dropped and returned no streams to ui.
Quick Questions/Suggestions:
1. When searching the tvseries type, i saw the prowlarr.py was issued the only imdbid , is there any way you can also pass the season and episode in query to prowlarr directly or the minimum pass the imdbid and season and then in prowalarr.py can match only the episode or range of episodes , as sometimes results or within in range ? As this would only fetch the exact results, so title matches can be ignored whenever the imdbid based search is sent out
2. when no imdbid based search query and search is free text type search , then matching title seems good , so we can filter out potential mismatches
Please forgive my ignorance as i fully did not understand the coding, my suggestions or questions are purely based on what i see on logs and how the returned results are working by observing.
INFO::16-Sep-24 19:16:29::prowlarr.py::394 - Found 15 streams for True Detective (2024) with tvsearch Search, params: {'query': '{IMDbId:tt2356777}', 'categories': [5000], 'type': 'tvsearch'}
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S03E02.720p.WEB.x265-MiNX[eztv].mkv'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S03E02.720p.WEB.x265-MiNX[TGx]'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S04E02.Night.Country.Parte.2.ITA.ENG.2160p.MAX.WEB-DL.DD5.1.HDR.DoVi.x265-MeM.GP.mkv'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S01E07.720p.HDTV.x264-KILLERS [PublicHD]'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S03E06.720p.WEB.H264-METCON[rarbg]'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S03E08.720p.WEB.x265-MiNX[TGx]'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S04E06.720p.10bit.WEBRip.2CH.x265.HEVC-PSA.mkv'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S04E04.1080p.WEB.H264-SuccessfulCrab'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S04E05.Night.Country.Part.5.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S03E05.720p.WEB.x265-MiNX[TGx]'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True Detective S01E07 720p HDTV ReEnc DeeJayAhmed'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S03E05.If.You.Have.Ghosts.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True Detective ( Season 1) 2014 BDRemux 1080p 3xUkr_Eng-Sub Ukr_Eng [Hurtom]'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S01E08.720p.rus.LostFilm.TV.mp4'
WARNING::16-Sep-24 19:16:29::base_scraper.py::181 - Title mismatch: 'Detective' vs. 'True Detective'. Torrent title: 'True.Detective.S02E01.PROPER.720p.HDTV.x264-KILLERS'
INFO::16-Sep-24 19:16:29::network.py::138 - Processed 15 items out of 15 total items.
INFO::16-Sep-24 19:16:29::prowlarr.py::275 - Finished processing 0 streams from 1 generators
INFO::16-Sep-24 19:16:29::prowlarr.py::93 - Returning 0 scraped streams for True Detective
INFO::16-Sep-24 19:16:29::utils.py::47 - Scraped 0 streams for True Detective
This was due to an issue in Parsett library https://github.com/dreulavelle/PTT the "True" keyword identified as audio=["TrueHd"]
>>> PTT.parse_title('True.Detective.S03E02.720p.WEB.x265-MiNX[eztv].mkv')
{'resolution': '720p', 'quality': 'WEB', 'codec': 'hevc', 'audio': ['TrueHD'], 'group': 'MiNX', 'container': 'mkv', 'seasons': [3], 'episodes': [2], 'extension': 'mkv', 'languages': [], 'title': 'Detective'}
cc: @dreulavelle
This was due to an issue in Parsett library https://github.com/dreulavelle/PTT the "True" keyword identified as audio=["TrueHd"]
>>> PTT.parse_title('True.Detective.S03E02.720p.WEB.x265-MiNX[eztv].mkv') {'resolution': '720p', 'quality': 'WEB', 'codec': 'hevc', 'audio': ['TrueHD'], 'group': 'MiNX', 'container': 'mkv', 'seasons': [3], 'episodes': [2], 'extension': 'mkv', 'languages': [], 'title': 'Detective'}
cc: @dreulavelle
But should we match title when imdbid and season is issued in query unless free text search? ,whereas result is exact match for title and seasons (atleast for sure the indexer bitmagnet i use seems to return exact result not sure about other indexers), episode returned sill could be in range or the whole season of episodes is given in collection as one result.
Not sure if the manifest.json option of parsing torrent vs not parsing torrent has anything to do with the above behaviour or it always only issues imdbid in query and always parses the title and stuff.
But should we match title when imdbid and season is issued in query unless free text search?
This is a bit risky. But I see the benefit in your case. Maybe I can bypass title check based on env config value
But should we match title when imdbid and season is issued in query unless free text search?
This is a bit risky. But I see the benefit in your case. Maybe I can bypass title check based on env config value
Thanks @mhdzumair ,I see your point ,it could behave differently with other indexers . Also ,I like idea of controlling through environment variable . Once you released the environment variable ,please update me ,I can test that 👍
Hi Team,
Thanks for developing the wonderful addon and sharing it to the users to self-host and use. I really thank and appreciate your efforts.
I have been using the self hosted bitmagnet torrent as my indexer configured with prowlarr. The integration works pretty fast and it works flawlessly for movies.
But whenever series is searched , it looks like mediafusion server errors out in the background with timing out on the front end with no results.
It errors out at line 682 in file Prowlarr.py where there is Year comparison .
ERROR LOG:
**
INFO::12-Sep-24 19:34:33 - Found 15 streams for The Umbrella Academy (4) (4) with IMDb ID tt1312171
INFO::12-Sep-24 19:34:33 - Processing 10 streams for The Umbrella Academy (4) (4) with IMDb ID
ERROR::12-Sep-24 19:34:35 - Unexpected error during batch processing: '<' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):
File "/mediafusion/utils/network.py", line 60, in call
File "/mediafusion/utils/network.py", line 47, in call
File "/mediafusion/scrapers/prowlarr.py", line 682, in parse_and_store_stream
TypeError: '<' not supported between instances of 'NoneType' and 'int'
ERROR::12-Sep-24 19:34:35 - Unexpected error during batch processing: '<' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):
File "/mediafusion/utils/network.py", line 60, in call
File "/mediafusion/utils/network.py", line 47, in call
File "/mediafusion/scrapers/prowlarr.py", line 682, in parse_and_store_stream
TypeError: '<' not supported between instances of 'NoneType' and 'int'
ERROR::12-Sep-24 19:34:35 - Unexpected error during batch processing: '<' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):
File "/mediafusion/utils/network.py", line 60, in call
File "/mediafusion/utils/network.py", line 47, in call
File "/mediafusion/scrapers/prowlarr.py", line 682, in parse_and_store_stream
TypeError: '<' not supported between instances of 'NoneType' and 'int'
ERROR::12-Sep-24 19:34:35 - Unexpected error during batch processing: '<' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):
File "/mediafusion/utils/network.py", line 60, in call
File "/mediafusion/utils/network.py", line 47, in call
File "/mediafusion/scrapers/prowlarr.py", line 682, in parse_and_store_stream
TypeError: '<' not supported between instances of 'NoneType' and 'int'
ERROR::12-Sep-24 19:34:35 - Unexpected error during batch processing: '<' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):
File "/mediafusion/utils/network.py", line 60, in call
File "/mediafusion/utils/network.py", line 47, in call
File "/mediafusion/scrapers/prowlarr.py", line 682, in parse_and_store_stream
TypeError: '<' not supported between instances of 'NoneType' and 'int'
INFO::12-Sep-24 19:34:38 - Retrying 5 items due to circuit breaker.
WARNING::12-Sep-24 19:34:48 - Timeout exceeded for operation: scrap_series_streams_from_prowlarr ('tt1312171', 'The Umbrella Academy', ['雨伞学院', 'Академия Амбрелла', 'Az Esernyő Akadémia', 'Skėčio akademija', 'The Umbrella Academy', 'Η ακαδημία της ομπρέλας', 'A Academia Umbrella', 'Академія Амбрелла', '伞学院'], 2019, 4, 4). Skipping.
INFO::12-Sep-24 19:34:48 - X.X.X.X - "GET http://media.mj.lan/***MASKED***/stream/series/tt1312171:4:4.json HTTP/1.1" 200
Front End with No Results:
Indexer config:
Mediafusion query to prowlarr with Search Result returned from Indexer: 15 results are shown
The result xml containing results:
I have also attached xml file containing results , so you can take a look.
series.results.output.log