search

rommapp / romm

A beautiful, powerful, self-hosted rom manager
https://romm.app
GNU Affero General Public License v3.0
1.76k stars 80 forks source link

[Bug] IGDB Scraper not working #104

Closed Fabaderheld closed 1 year ago

Fabaderheld commented 1 year ago

Hi,

I looked a round in the issues but couldn't find anything related to my problem. I played around with romm the last couple of days but I couldn't get the IGDB scraper to work. For every title I search get no results and the logs show this error:

 INFO:     [ROMM] getting Arkanoid.d64 roms from  igdb ... 2023-04-01 13:38:03
INFO:     [ROMM] Matched roms for Arkanoid.d64: [{'title': 'Syntax Error', 'status': 400, 'cause': "Expecting a STRING as input, surround your input with quotes starting at ']' expecting {'{', 'f', '(', '[', 'true', 't', 'false', 'null', 'n'"}] 2023-04-01 13:38:03
INFO:     192.168.0.6:0 - "PUT /search/roms/igdb HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/back/./main.py", line 101, in search_rom_igdb
    return {'data': igdbh.get_matched_roms(data['file_name'], data['p_igdb_id'])}
  File "/back/./handler/igdb_handler.py", line 28, in wrapper
    return func(*args)
  File "/back/./handler/igdb_handler.py", line 125, in get_matched_roms
    data=f"fields url; where game={rom['id']};").json()[0]
KeyError: 'id'

The IGDB client_id and secret are working when call in the API with powershell. I'm running version 1.6 in Docker.

Thanks

Faba

zurdi15 commented 1 year ago

Hello, can you share a screenshot or a stringlike of your folder structure?

Fabaderheld commented 1 year ago 
.
├── C64
│   ├── Emu
│   └── Roms

like this. And its's the same for all systems

zurdi15 commented 1 year ago

Hi! your issue is solved with the last release v1.6.2 for two reasons:

EDIT: make sure to adapt the docker-compose file to the new version

Fabaderheld commented 1 year ago

Thanks, but this was just an example of the error message. I have with any rom on any system. Maybe be setup is wrong?

GB:

INFO:     [ROMM] Matched roms for Alien vs Predator - The Last of His Clan (USA).gb: [{'title': 'Syntax Error', 'status': 400, 'cause': "Expecting a STRING as input, surround your input with quotes starting at ']' expecting {'{', 'f', '(', '[', 'true', 't', 'false', 'null', 'n'"}] 2023-04-04 16:33:37
INFO:     192.168.0.6:0 - "PUT /search/roms/igdb HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/back/./main.py", line 101, in search_rom_igdb
    return {'data': igdbh.get_matched_roms(data['file_name'], data['p_igdb_id'])}
  File "/back/./handler/igdb_handler.py", line 28, in wrapper
    return func(*args)
  File "/back/./handler/igdb_handler.py", line 125, in get_matched_roms
    data=f"fields url; where game={rom['id']};").json()[0]
KeyError: 'id'

NES:

INFO:     [ROMM] Matched roms for Super Mario Bros 3 [p1].nes: [{'title': 'Syntax Error', 'status': 400, 'cause': "Expecting a STRING as input, surround your input with quotes starting at ']' expecting {'{', 'f', '(', '[', 'true', 't', 'false', 'null', 'n'"}] 2023-04-04 16:35:24
INFO:     192.168.0.6:0 - "PUT /search/roms/igdb HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/back/./main.py", line 101, in search_rom_igdb
    return {'data': igdbh.get_matched_roms(data['file_name'], data['p_igdb_id'])}
  File "/back/./handler/igdb_handler.py", line 28, in wrapper
    return func(*args)
  File "/back/./handler/igdb_handler.py", line 125, in get_matched_roms
    data=f"fields url; where game={rom['id']};").json()[0]
KeyError: 'id'

SNES:

INFO:     [ROMM] Matched roms for ActRaiser (USA).sfc: [{'title': 'Syntax Error', 'status': 400, 'cause': "Expecting a STRING as input, surround your input with quotes starting at ']' expecting {'{', 'f', '(', '[', 'true', 't', 'false', 'null', 'n'"}] 2023-04-04 16:35:53
INFO:     192.168.0.6:0 - "PUT /search/roms/igdb HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/back/./main.py", line 101, in search_rom_igdb
    return {'data': igdbh.get_matched_roms(data['file_name'], data['p_igdb_id'])}
  File "/back/./handler/igdb_handler.py", line 28, in wrapper
    return func(*args)
  File "/back/./handler/igdb_handler.py", line 125, in get_matched_roms
    data=f"fields url; where game={rom['id']};").json()[0]
KeyError: 'id'

I don't know if this helps, but I don't see any system icons on the left side.

Thanks

Faba

Fabaderheld commented 1 year ago

My guess is that it is case sensitiv, I tried it again but ran into #127

zurdi15 commented 1 year ago

Ohh okay, I can see now with the rest of the logs. You are true, platforms folders are case-sensitive. But in any case, with the v1.6.2 version, even if you have the platforms folder uppercase should not fail, just not find anything. Make them lowercase and tell me if it works

Fabaderheld commented 1 year ago

yeah my bad, those were the errors from the old version.

In 1.6.2 I'm getting:

WARNING:  [ROMM] SNES is not supported! 2023-04-04 16:38:44
zurdi15 commented 1 year ago

So yes, as we suspected its because the case-sensitive. I close the issue but will add the no-case-sensitive platforms folders in the near future.

Thank you for the feedback!