[Bug] Scan always stops at the same point

[paparapapapapa]

RomM version Can't find it but unRaid said container is up to date.

Describe the bug Romm stops searching the library after raising some exception. UI effects keep on like if it was still searching, no error on the UI. I tried multiple time, purging img, container and appdata to no avail. Also, my library has many platforms and starts with 3do, arcade, ... but everytime Romm starts from gb instead, then stops somewhere during scanning the gba roms. So I always see only gb and gba as platforms in the UI. My folder structure is of type B, because is compatible with EmulatorJS, running also on unRaid without issues.

To Reproduce Steps to reproduce the behavior:

1. Install and configure a clean Romm image
2. Click on Scan
3. Wait and see error on logs

Logs

INFO: [nginx][2024-02-22 10:05:30] 192.168.12.208 - - "POST /api/tasks/run HTTP/1.1" 500 21 "http://192.168.55.6:8093/settings/control-panel" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.3.1 Safari/605.1.15" rt=0.006 uct="0.000" uht="0.006" urt="0.006" INFO: None - "WebSocket /socket.io/?EIO=4&transport=websocket" [accepted] INFO: connection open INFO: [RomM][2024-02-22 10:07:06] 🔎 Scanning INFO: [RomM][2024-02-22 10:07:08] Found 39 platforms in file system INFO: [RomM][2024-02-22 10:07:08] · gb INFO: [RomM][2024-02-22 10:07:08] Identified as Game Boy 🎮 WARNING: [RomM][2024-02-22 10:07:08] 8 roms found [...] INFO: [RomM][2024-02-22 10:10:39] Identified as Grand Theft Auto Advance 👾 INFO: [RomM][2024-02-22 10:10:40] · [GBA] Grand Theft Auto Advance [ITA].gba 2 Task exception was never retrieved future: <Task finished name='Task-34' coro=<AsyncServer._handle_event_internal() done, defined at /backend/lib/python3.10/site-packages/socketio/asyncio_server.py:522> exception=AttributeError("'NoneType' object has no attribute 'group'")> Traceback (most recent call last): File "/backend/lib/python3.10/site-packages/socketio/asyncio_server.py", line 524, in _handle_event_internal r = await server._trigger_event(data[0], namespace, sid, data[1:]) File "/backend/lib/python3.10/site-packages/socketio/asyncio_server.py", line 558, in _trigger_event ret = await handler(args) File "/backend/endpoints/scan.py", line 120, in scan_handler await scan_platforms(platform_slugs, complete_rescan, rescan_unidentified, selected_roms) File "/backend/endpoints/scan.py", line 74, in scan_platforms scanned_rom = await scan_rom(scanned_platform, fs_rom) File "/backend/utils/fastapi.py", line 84, in scan_rom "file_extension": get_file_extension(rom_attrs), File "/backend/utils/init.py", line 129, in get_file_extension re.search(EXTENSION_REGEX, rom["file_name"]).group(1) AttributeError: 'NoneType' object has no attribute 'group'

Screenshots Mappings:

Server:

• OS: unRaid 6.12.8 Pro
• Docker version: 24.0.9
• Browser: Safari

Config file is like default, I just added these two rows at the end:

sega32x: 'sega32' segaGG: 'gamegear'

Additional context Add any other context about the problem here.

[paparapapapapa]

I deleted the file [GBA] Grand Theft Auto Advance [ITA].gba 2, restarted the container and launched another complete search. Now It silently fails there:

[...]
INFO:     [RomM][2024-02-22 11:04:09]    · ind-mdit.gba 2
Task exception was never retrieved
future: <Task finished name='Task-22' coro=<AsyncServer._handle_event_internal() done, defined at /backend/lib/python3.10/site-packages/socketio/asyncio_server.py:522> exception=AttributeError("'NoneType' object has no attribute 'group'")>
Traceback (most recent call last):
  File "/backend/lib/python3.10/site-packages/socketio/asyncio_server.py", line 524, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
  File "/backend/lib/python3.10/site-packages/socketio/asyncio_server.py", line 558, in _trigger_event
    ret = await handler(*args)
  File "/backend/endpoints/scan.py", line 120, in scan_handler
    await scan_platforms(platform_slugs, complete_rescan, rescan_unidentified, selected_roms)
  File "/backend/endpoints/scan.py", line 74, in scan_platforms
    scanned_rom = await scan_rom(scanned_platform, fs_rom)
  File "/backend/utils/fastapi.py", line 84, in scan_rom
    "file_extension": get_file_extension(rom_attrs),
  File "/backend/utils/__init__.py", line 129, in get_file_extension
    re.search(EXTENSION_REGEX, rom["file_name"]).group(1)
AttributeError: 'NoneType' object has no attribute 'group'

I never noticed these "2" at the end of the extension of some files, I don't know if they were added by some batch process, like extraction from compressed archives, anyways it seems like you just need to catch this exception and show an error?

I will batch delete those files and then report back.

[gantoine]

anyways it seems like you just need to catch this exception and show an error?

yes right now we don't do a great job of showing scan errors in the UI, that'll have to be improved. you're right that the file extensions are the issue, romm expects valid file extensions and obviously .gba 2 isn't one.

[paparapapapapa]

Ok I can confirm that was the issue. This morning I woke up and found the library updated.

I would ask you another question here if you don't mind, otherwise I will open another issue.

As I mentioned above, I added two rows to the config file, but this morning I found the following issues:

So I added others. Now they look like this:

system:
  platforms:
    # [your custom platform folder name]: [RomM platform name]
    gc: 'ngc'
    psx: 'ps'
    sega32x: 'sega32'
    segaGG: 'gamegear'
    ngp: 'neo-geo-pocket-color'
    pce: 'turbografx-16-slash-pc-engine-cd'
    segamd: 'genesis-slash-megadrive'
    segams: 'sms'

And now I get:

[...]
INFO:     [RomM][2024-02-23 08:26:53] · ngp
INFO:     [RomM][2024-02-23 08:26:53]   Identified as Neo Geo Pocket Color 🎮
Task exception was never retrieved
future: <Task finished name='Task-24' coro=<AsyncServer._handle_event_internal() done, defined at /backend/lib/python3.10/site-packages/socketio/asyncio_server.py:522> exception=IntegrityError('(sqlite3.IntegrityError) UNIQUE constraint failed: platforms.fs_slug')>
Traceback (most recent call last):
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: platforms.fs_slug

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/backend/lib/python3.10/site-packages/socketio/asyncio_server.py", line 524, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
  File "/backend/lib/python3.10/site-packages/socketio/asyncio_server.py", line 558, in _trigger_event
    ret = await handler(*args)
  File "/backend/endpoints/scan.py", line 120, in scan_handler
    await scan_platforms(platform_slugs, complete_rescan, rescan_unidentified, selected_roms)
  File "/backend/endpoints/scan.py", line 47, in scan_platforms
    _new_platform = dbh.add_platform(scanned_platform)
  File "/backend/handler/db_handler.py", line 26, in wrapper
    with args[0].session.begin() as s:
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1755, in _maker_context_manager
    with self.begin():
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/util.py", line 146, in __exit__
    with util.safe_reraise():
  File "/backend/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/util.py", line 144, in __exit__
    self.commit()
  File "<string>", line 2, in commit
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1256, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1231, in _prepare_impl
    self.session.flush()
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4312, in flush
    self._flush(objects)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4447, in _flush
    with util.safe_reraise():
  File "/backend/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4408, in _flush
    flush_context.execute()
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1043, in _emit_insert_statements
    result = connection.execute(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
    ret = self._execute_context(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
    return self._exec_single_context(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
    self._handle_dbapi_exception(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
    self.dialect.do_execute(
  File "/backend/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: platforms.fs_slug
[SQL: INSERT INTO platforms (slug, fs_slug, name, igdb_id, sgdb_id, logo_path, n_roms) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('neo-geo-pocket-color', 'ngp', 'Neo Geo Pocket Color', 120, None, 'default/default/cover/small.png', 0)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

I suspect this is because of wrong [RomM platform name]. I used the ones I found in your icon set.

[gantoine]

Your config is right, but you'll need to do some cleanup. Head into each of the galleries for platforms that didn't match, in the top-right there's a 3-dot menu, "delete" the platform (don't worry it won't delete your games), then run another scan with the updated config. Just make sure you restart romm after each config update (there's a bug that'll be fixed in 3.0).

[paparapapapapa]

Ah ok, now it's working fine, except for

    segamd: 'genesis-slash-megadrive'
    segams: 'sms'

that return Not found in IGDB

[gantoine]

So all the others in your config are working except for those two?

[paparapapapapa]

Yes that's correct. This time the process doesn't fail, it just doesn't complete and shows the warning marks on the UI. I tried another time to follow your last indications, just to be sure, but nothing changed.