search

lheyberger / mtg-parser

Magic: the Gathering decklist parser
MIT License
10 stars 1 forks source link

Moxfield scraping is against TOS #9

Closed Dimfred closed 4 months ago

Dimfred commented 4 months ago

Hi just tried .moxfield and it seems that it is against the tos.

Log:

2024-05-02 08:19:50.365 | ERROR    | magic_circle.app:handle_internal_general_error:89 - 500: !!!!!!!!!!!!!!!!!! UNHANDLED !!!!!!!!!!!!!!!!!!
2024-05-02 08:19:50.366 | ERROR    | magic_circle.app:handle_internal_general_error:90 - Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):

  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           │           │     │    │       └ {}
           │           │     │    └ <property object at 0x77fc9209f290>
           │           │     └ <Response [403]>
           │           └ <function loads at 0x77fcc05e6200>
           └ <module 'json' from '/usr/lib/python3.11/json/__init__.py'>
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           │                │      └ 'Scraping this website is against the Terms of Service. Please contact support@moxfield.com.'
           │                └ <function JSONDecoder.decode at 0x77fcc05e5b20>
           └ <json.decoder.JSONDecoder object at 0x77fcc05e8290>
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               │    │          │      │  └ 'Scraping this website is against the Terms of Service. Please contact support@moxfield.com.'
               │    │          │      └ <built-in method match of re.Pattern object at 0x77fcc05c4c70>
               │    │          └ 'Scraping this website is against the Terms of Service. Please contact support@moxfield.com.'
               │    └ <function JSONDecoder.raw_decode at 0x77fcc05e5bc0>
               └ <json.decoder.JSONDecoder object at 0x77fcc05e8290>
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
          │                                  └ 'Scraping this website is against the Terms of Service. Please contact support@moxfield.com.'
          └ <class 'json.decoder.JSONDecodeError'>

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "<string>", line 1, in <module>
  File "/usr/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               │     │   └ 4
               │     └ 7
               └ <function _main at 0x77fcc0ff2660>
  File "/usr/lib/python3.11/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
           │    │          └ 4
           │    └ <function BaseProcess._bootstrap at 0x77fcc10cfce0>
           └ <SpawnProcess name='SpawnProcess-20' parent=411669 started>
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    │    └ <function BaseProcess.run at 0x77fcc10cf240>
    └ <SpawnProcess name='SpawnProcess-20' parent=411669 started>
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {'config': <uvicorn.config.Config object at 0x77fcbf8b1050>, 'target': <bound method Server.run of <uvicorn.server.Server obj...
    │    │        │    │        └ <SpawnProcess name='SpawnProcess-20' parent=411669 started>
    │    │        │    └ ()
    │    │        └ <SpawnProcess name='SpawnProcess-20' parent=411669 started>
    │    └ <function subprocess_started at 0x77fcc0210b80>
    └ <SpawnProcess name='SpawnProcess-20' parent=411669 started>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 78, in subprocess_started
    target(sockets=sockets)
    │              └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8000)>]
    └ <bound method Server.run of <uvicorn.server.Server object at 0x77fcbf892d50>>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
           │       │   │    │             └ [<socket.socket fd=3, family=2, type=1, proto=0, laddr=('0.0.0.0', 8000)>]
           │       │   │    └ <function Server.serve at 0x77fcc03e7b00>
           │       │   └ <uvicorn.server.Server object at 0x77fcbf892d50>
           │       └ <function run at 0x77fcc0ce2fc0>
           └ <module 'asyncio' from '/usr/lib/python3.11/asyncio/__init__.py'>
  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           │      │   └ <coroutine object Server.serve at 0x77fcc00595d0>
           │      └ <function Runner.run at 0x77fcc0571800>
           └ <asyncio.runners.Runner object at 0x77fcc0f5a210>
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           │    │     │                  └ <Task pending name='Task-1' coro=<Server.serve() running at /home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/...
           │    │     └ <function BaseEventLoop.run_until_complete at 0x77fcc056b420>
           │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
           └ <asyncio.runners.Runner object at 0x77fcc0f5a210>
  File "/usr/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
    self.run_forever()
    │    └ <function BaseEventLoop.run_forever at 0x77fcc056b380>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
    self._run_once()
    │    └ <function BaseEventLoop._run_once at 0x77fcc05711c0>
    └ <_UnixSelectorEventLoop running=True closed=False debug=False>
  File "/usr/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once
    handle._run()
    │      └ <function Handle._run at 0x77fcc0ce37e0>
    └ <Handle Task.task_wakeup(<Future finished result=None>)>
  File "/usr/lib/python3.11/asyncio/events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
    │    │            │    │           │    └ <member '_args' of 'Handle' objects>
    │    │            │    │           └ <Handle Task.task_wakeup(<Future finished result=None>)>
    │    │            │    └ <member '_callback' of 'Handle' objects>
    │    │            └ <Handle Task.task_wakeup(<Future finished result=None>)>
    │    └ <member '_context' of 'Handle' objects>
    └ <Handle Task.task_wakeup(<Future finished result=None>)>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
                   └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x77fc91d1fa10>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
                 │    │   │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
                 │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
                 │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
                 │    └ <fastapi.applications.FastAPI object at 0x77fc9008e7d0>
                 └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x77fc91d1fa10>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
                           │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
                           │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
                           └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │                │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x77fc91d224d0>
          └ <fastapi.applications.FastAPI object at 0x77fc9008e7d0>
> File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
          │    │   │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x77fc924868e0>
          │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.cors.CORSMiddleware object at 0x77fc921040d0>
          └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x77fc91d224d0>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
          │    │               │      │        │                     └ Headers({'host': 'localhost:8000', 'connection': 'keep-alive', 'content-length': '163', 'sec-ch-ua': '"Google Chrome";v="123"...
          │    │               │      │        └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x77fc924868e0>
          │    │               │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │               └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <function CORSMiddleware.simple_response at 0x77fc930f4f40>
          └ <starlette.middleware.cors.CORSMiddleware object at 0x77fc921040d0>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response
    await self.app(scope, receive, send)
          │    │   │      │        └ functools.partial(<bound method CORSMiddleware.send of <starlette.middleware.cors.CORSMiddleware object at 0x77fc921040d0>>, ...
          │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x77fc91d21090>
          └ <starlette.middleware.cors.CORSMiddleware object at 0x77fc921040d0>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
          │                            │    │    │     │      │        └ functools.partial(<bound method CORSMiddleware.send of <starlette.middleware.cors.CORSMiddleware object at 0x77fc921040d0>>, ...
          │                            │    │    │     │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │                            │    │    │     └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │                            │    │    └ <starlette.requests.Request object at 0x77fc8ff31050>
          │                            │    └ <fastapi.routing.APIRouter object at 0x77fcbf71ad50>
          │                            └ <starlette.middleware.exceptions.ExceptionMiddleware object at 0x77fc91d21090>
          └ <function wrap_app_handling_exceptions at 0x77fc93efbce0>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
          │   │      │        └ <function wrap_app_handling_exceptions.<locals>.wrapped_app.<locals>.sender at 0x77fc8ff227a0>
          │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          └ <fastapi.routing.APIRouter object at 0x77fcbf71ad50>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
          │    │                │      │        └ <function wrap_app_handling_exceptions.<locals>.wrapped_app.<locals>.sender at 0x77fc8ff227a0>
          │    │                │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │                └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <bound method Router.app of <fastapi.routing.APIRouter object at 0x77fcbf71ad50>>
          └ <fastapi.routing.APIRouter object at 0x77fcbf71ad50>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
          │     │      │      │        └ <function wrap_app_handling_exceptions.<locals>.wrapped_app.<locals>.sender at 0x77fc8ff227a0>
          │     │      │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │     │      └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │     └ <function Route.handle at 0x77fc93f21300>
          └ APIRoute(path='/cards', name='get_cards', methods=['POST'])
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
          │    │   │      │        └ <function wrap_app_handling_exceptions.<locals>.wrapped_app.<locals>.sender at 0x77fc8ff227a0>
          │    │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │    │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │    └ <function request_response.<locals>.app at 0x77fc90082160>
          └ APIRoute(path='/cards', name='get_cards', methods=['POST'])
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
          │                            │    │        │      │        └ <function wrap_app_handling_exceptions.<locals>.wrapped_app.<locals>.sender at 0x77fc8ff227a0>
          │                            │    │        │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │                            │    │        └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          │                            │    └ <starlette.requests.Request object at 0x77fc8ff30f90>
          │                            └ <function request_response.<locals>.app.<locals>.app at 0x77fc8ff20360>
          └ <function wrap_app_handling_exceptions at 0x77fc93efbce0>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
          │   │      │        └ <function wrap_app_handling_exceptions.<locals>.wrapped_app.<locals>.sender at 0x77fc90081940>
          │   │      └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.h11_impl.RequestResponseCycle object at 0x77fc8ff3e3d0>>
          │   └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.4'}, 'http_version': '1.1', 'server': ('127.0.0.1', 8000), 'cl...
          └ <function request_response.<locals>.app.<locals>.app at 0x77fc8ff20360>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
                     │    └ <starlette.requests.Request object at 0x77fc8ff30f90>
                     └ <function get_request_handler.<locals>.app at 0x77fc90082020>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                         └ <function run_endpoint_function at 0x77fc93f20e00>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
                 │         │      └ {'repo': <sqlalchemy.orm.session.Repository object at 0x77fc8fdfd290>, 'user': UserDB(username='theo', id=50, password='8c697...
                 │         └ <function get_cards at 0x77fc90080ea0>
                 └ <fastapi.dependencies.models.Dependant object at 0x77fc8fecdf10>

  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/magic_circle/card/routes.py", line 56, in get_cards
    cards = parser.parse_decklist(req.decklist)
            │      │              │   └ 'https://www.moxfield.com/decks/gR0IqrZXW0CRKzld_kQ3kw'
            │      │              └ CardsGetIn(page=0, max_page=50, usernames=['admin', 'admin2', 'theo', 'aasdf'], decklist='https://www.moxfield.com/decks/gR0I...
            │      └ <function PlainParser.parse_decklist at 0x77fc900707c0>
            └ <magic_circle.card.parser.PlainParser object at 0x77fc91d1d5d0>

  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/magic_circle/card/parser.py", line 86, in parse_decklist
    decklist_str = mtg_parser.moxfield.parse_deck(decklist_str)  # type: ignore
                   │          │        │          └ 'https://www.moxfield.com/decks/gR0IqrZXW0CRKzld_kQ3kw'
                   │          │        └ <function parse_deck at 0x77fc91faa980>
                   │          └ <module 'mtg_parser.moxfield' from '/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-pack...
                   └ <module 'mtg_parser' from '/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/mtg_...

  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/mtg_parser/moxfield.py", line 28, in parse_deck
    deck = _parse_deck(_download_deck(src, session))
           │           │              │    └ <module 'requests' from '/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/reques...
           │           │              └ 'https://www.moxfield.com/decks/gR0IqrZXW0CRKzld_kQ3kw'
           │           └ <function _download_deck at 0x77fc91fa3100>
           └ <function _parse_deck at 0x77fc91fa3380>
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/mtg_parser/moxfield.py", line 36, in _download_deck
    return session.get(url).json()
           │       │   └ 'https://api.moxfield.com/v2/decks/all/gR0IqrZXW0CRKzld_kQ3kw'
           │       └ <function get at 0x77fc920798a0>
           └ <module 'requests' from '/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/reques...
  File "/home/dimfred/workspaces/magic-circle/magic-circle-backend/.venv/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
          └ <class 'requests.exceptions.JSONDecodeError'>
import mtg_parser

mtg_parser.moxfield.parse_deck("https://www.moxfield.com/decks/gR0IqrZXW0CRKzld_kQ3kw")
lheyberger commented 4 months ago

This is a known issue: https://github.com/lheyberger/mtg-parser?tab=readme-ov-file#moxfield You can override the requests.Session that is used by the library internally with your own custom Session (you can look at the requests_session in the ./tests/utils.py for inspiration).

Dimfred commented 4 months ago

Sry as always, haven't checked the readme thx, closing.