No games showing on the web library

shoyrock commented 1 month ago

Hi, I'm a noob here. I was using v2 last year, and it worked for the most part. I updated it some months ago, and it broke the install, so I wasn't able to use it anymore. I removed everything from Docker and did a clean install. The app seems to be working now, but nothing is showing up in the library when I start the web UI. I'm still able to see the game on Tinfoil, so I know I'm using the right path. Screenshot 2024-07-27 185651

a1ex4 commented 1 month ago

Honestly it may very well be the size of library that's causing the issue on the web part, I'm only testing with 10% of what you have.

When you access the web UI the whole database content is sent to the web page, so I can handle all the filtering, pagination and different display on the front end, client side. But maybe it is simply not a good solution with a larger library, and I have to go back to filtering on the backend and only displaying in the front end.

shoyrock commented 1 month ago

Thanks for the information. Also, I found that I am unable to change the welcome message; I don’t know if it’s related, but just thought I would mention it

Terror-Gene commented 1 month ago

i did have the page working earlier today, but after adding more of my games to the folder it stopped adding them, i thought they weren't being processed, but after a while i came back & now it displays nothing. when it was showing games it maxed out at 27 (with base filter on), i rescanned a few times & it only showed 27. but now nothing (there are 77 games in the folder). & i also cant change the motd. i instead set the message in the settings.yaml file in /app/config

shoyrock commented 1 month ago

i did have the page working earlier today, but after adding more of my games to the folder it stopped adding them, i thought they weren't being processed, but after a while i came back & now it displays nothing. when it was showing games it maxed out at 27 (with base filter on), i rescanned a few times & it only showed 27. but now nothing (there are 77 games in the folder). & i also cant change the motd. i instead set the message in the settings.yaml file in /app/config

Thanks, I was able to get the shop name using the settings.yaml. the game list was working last year don't know what point is was broken

a1ex4 commented 1 month ago

So from the reports of everyone it seems it is not the library size that's causing the UI to be empty. My guess is that some file is causing the identification / scraping process to fail, and the backend has an error and just returns nothing. Nothing in the logs is displayed? There are no errors or exception?

The biggest way you could help me is by sharing your ownfoil.db file, so I can replicate your context and see exactly why there's nothing showing up.

Side note, this kind of behavior tells me that the two things to be implemented with higher priority is proper logging everywhere (that will hopefully fix the issue where not all log lines are displayed in the container logs), and the toast notifications for every failed API call to clearly indicate to the user the errors that happened, instead of having a blank page.

shoyrock commented 1 month ago

So from the reports of everyone it seems it is not the library size that's causing the UI to be empty. My guess is that some file is causing the identification / scraping process to fail, and the backend has an error and just returns nothing. Nothing in the logs is displayed? There are no errors or exception?

The biggest way you could help me is by sharing your ownfoil.db file, so I can replicate your context and see exactly why there's nothing showing up.

Side note, this kind of behavior tells me that the two things to be implemented with higher priority is proper logging everywhere (that will hopefully fix the issue where not all log lines are displayed in the container logs), and the toast notifications for every failed API call to clearly indicate to the user the errors that happened, instead of having a blank page.

ownfoil.db.zip here you go

shoyrock commented 1 month ago

def generate_library(): titles = get_all_titles_from_db() games_info = []

for title in titles:
    if title['type'] == APP_TYPE_UPD:
        continue

    info_from_titledb = get_game_info(title['app_id'])
    if info_from_titledb is None:
        print(f'Info not found for game: {title}')
        continue

    title.update(info_from_titledb)

    if title['type'] == APP_TYPE_BASE:
        title.update(get_library_status(title['app_id']))
        title['title_id_name'] = title['name']

    elif title['type'] == APP_TYPE_DLC:
        process_dlc(title)

    games_info.append(title)

return sorted(games_info, key=lambda x: (x.get("title_id_name", "Unrecognized"), x.get('app_id', "")))

def process_dlc(title): all_dlc_versions = get_all_dlc_existing_versions(title['app_id']) app_id_version_from_txt = get_app_id_version_from_versions_txt(title['app_id'])

title['has_latest_version'] = (
    (all_dlc_versions and title['version'] == all_dlc_versions[-1]) or
    (app_id_version_from_txt and title['version'] == int(app_id_version_from_txt))
)

titleid_info = get_game_info(title['title_id'])
title['title_id_name'] = titleid_info.get('name', 'Unknown')

shoyrock commented 1 month ago

I just switched over to :develop idk if some just fix the problem or maybe the problem is just in :v2

a1ex4 commented 1 month ago

I just switched over to :develop idk if some just fix the problem or maybe the problem is just in :v2

Yeah I pushed some bug fixes to develop version, I saw some errors using your db, thanks. Do you confirm it's fixed?

Terror-Gene commented 1 month ago

can confirm the webpage shows all games now (not using :develop). thanks!

on an unrelated note, i noticed high cpu usage on this container last night & my servers ram was at 97%, after stopping this container my ram dropped to 47% & cpu usage stopped. (have 64GB ram). not sure what the issue was since there is little debug. but thought id mention it. will make an issue if the problem happens again.

shoyrock commented 1 month ago

Yeah I pushed some bug fixes to develop version, I saw some errors using your db, thanks. Do you confirm it's fixed?

Yes, I can confirm that it's working now. this is all the errors that I seeing after this bugfix Identifiying file : /Pokémon Shining Pearl[010018E011D92000][v0].nsz Could not identify file /games/Pokémon Shining Pearl[010018E011D92000][v0].nsz from metadata: 'NoneType' object has no attribute 'lower'. Trying identification with filename... No files were deleted. All files are present on disk. DLC app ID not in cnmts.json: 01004BA017CD7011 Title ID not found in titledb: 01004BA017CD7000 DLC app ID not in cnmts.json: 01004BA017CD7012 Title ID not found in titledb: 01004BA017CD7000 DLC app ID not in cnmts.json: 01004BA017CD7013 Title ID not found in titledb: 01004BA017CD7000 DLC app ID not in cnmts.json: 01004BA017CD7014 Title ID not found in titledb: 01004BA017CD7000 DLC app ID not in cnmts.json: 01004BA017CD7016 Title ID not found in titledb: 01004BA017CD7000 DLC app ID not in cnmts.json: 01004BA017CD7019 Title ID not found in titledb: 01004BA017CD7000 DLC app ID not in cnmts.json: 0100D1501ABAF001 Title ID not found in titledb: 0100D1501ABAF000 DLC app ID not in cnmts.json: 0100D1501ABAF002 Title ID not found in titledb: 0100D1501ABAF000 DLC app ID not in cnmts.json: 0100D1501ABAF004 Title ID not found in titledb: 0100D1501ABAF000 DLC app ID not in cnmts.json: 0100D1501ABAF003 Title ID not found in titledb: 0100D1501ABAF000 DLC app ID not in cnmts.json: 01006560184E7005 Title ID not found in titledb: 01006560184E7000 DLC app ID not in cnmts.json: 01006560184E7004 Title ID not found in titledb: 01006560184E7000 DLC app ID not in cnmts.json: 01006560184E7005 Title ID not found in titledb: 01006560184E7000 DLC app ID not in cnmts.json: 01006560184E700A Title ID not found in titledb: 01006560184E7000 Title ID not found in titledb: 01006560184E7099 DLC app ID not in cnmts.json: 01006560184E7099 Title ID not found in titledb: 01006560184E7000 DLC app ID not in cnmts.json: 01006560184E7006 Title ID not found in titledb: 01006560184E7000 DLC app ID not in cnmts.json: 01006560184E700C Title ID not found in titledb: 01006560184E7000 DLC app ID not in cnmts.json: 0100CF401A98F001 Title ID not found in titledb: 0100CF401A98F000 DLC app ID not in cnmts.json: 010073F0189B7001 Title ID not found in titledb: 010073F0189B7000 DLC app ID not in cnmts.json: 0100F7E00C70F005 Title ID not found in titledb: 0100F7E00C70F000 DLC app ID not in cnmts.json: 0100210019429003 Title ID not found in titledb: 0100210019429000 DLC app ID not in cnmts.json: 0100D1501ABAF007 Title ID not found in titledb: 0100D1501ABAF000

a1ex4 commented 1 month ago

Thank you both for the feedback, I'll close the issue then.

For the higher than expected resource usage it is true that there is a lot of room for optimization. Most of the implementation design was "make it work as fast as possible" to at least have the basic features implemented, and test everything. I'll come around to optimize the memory usage for sure.

As for the remaining "errors" I'll add a view in the UI to summarize all the files which could not be identified to troubleshoot.

a1ex4 / ownfoil

No games showing on the web library #88