RAWG integration cannot find properly

[nodiaque]

Hello,

I've installed the back-end on unraid with the database and enabled all store. I add the game with the exact name from RAWG, let's say Hellbound. When I get into the client, I see it has downloaded first image from somewhere on the internet and later, it has a RAWG refresh. The name change for something else it and sync with anything.

I added 40 games and it did that for all 40 games. I normally have the right images (sometime not) but the RAWG sync always sync with something else.

[Alfagun74]

Is only the image wrong or all data?

Can you show me your .env file or configuration?

Are you aware of this?

Thanks in advance.

[nodiaque]

Getting back with the file as soon as I get back home. As for the manual editing, yes I'm aware, that's why I said I have to manually modify all of them one by one.

[nodiaque]

ok for the env file, I have the default one cause there's nothing in my gamevault folder except my image cache. I do have /data mounted but nothing in it. Weird?

I have environment variable from unraid template RAWG_API_INCLUDED_STORES: = 0

I just tried a new game, Infinity Wars

Here's the result, notice how RAWG tagger to anything

Here's the file

RAWG https://rawg.io/games/infinity-wars

And another game, HeroQuest II: Legacy of Sorasil

https://rawg.io/games/heroquest-ii-legacy-of-sorasil

This time, it seems the image are good. But it's the always the case. Often, it will get the good background but the box art is not good. I think it just didn't refresh yet.

[Alfagun74]

Try setting RAWG_API_INCLUDED_STORES to the default 1,2,3,4,5,6,7,8,11 and see if it still is this bad. This excludes a plethora of itch.io Games, making the search more accurate.

[nodiaque]

Hello, it was on the default before and it was worst because lots of game would simply not be found even with manual search. Games like Heroes of Might and Magic 4 Winds of War for exemple.

I don't know how the search is done, but right now, what it find is very different from the search. Just HeroQuest 2, the only matching word are "2" and "of". In the manual search, when I input exactly the title, it's the first return.

[Alfagun74]

This is horrible, yes.

Can set the log level to debug, index about 5-10 new games and send a full log file so i can analyze?

[nodiaque]

Ok, I've done it. Where can I find the file on the docker? Cause there's nothing in /data, I'm wondering why this is mapped...

edit: nevermind, found the env value for it. Also, found out that unraid create a /data for nothing

[nodiaque]

Here's the log, and I'm amazed at how it find nothing!

Check for the following game: Police Quest Collection ==> Matched as Falcon Collection Risk 2 ==> Matched as Grand Turismo 2 Prototype => Matched as Prototype! (A first!) Zombie In My Pocket New ==> There's A Zombie in my Closet (and image is Zombie Ate My neighboor) Space Quest 2 Vohaul's Revenge ==> King's Quest1+2+3

Weird thing is looking at the log file, none have the actual game in the game list it find so something is very wrong. Unless the search doesn't use the "all store" like the main search? gamevault-backend-2024-03-12.log

[Alfagun74]

Let me analyze this thoroughly and come back to you asap.

[Alfagun74]

I just checked the logs and the acurracy is truly horrible! Almost looks like it takes page 4 or something. Never seen this behaviour before. Let me try to reproduce it. May i ask what country you are located in?

[Alfagun74]

I can not reproduce this unfortunately. Note the Config in the Logs. Stores=0.

(I changed the logs so they look a bit better, but the application logic is the same)

gamevault-backend-2024-03-13.log.json

Can you try and check if the unstable v10.2.1 Pre-Release image behaves any different for you?

[nodiaque]

I just checked the logs and the acurracy is truly horrible! Almost looks like it takes page 4 or something. Never seen this behaviour before. Let me try to reproduce it. May i ask what country you are located in?

I'm in Canada.

Could it be that the env in the docker doesn't say to search for all store? Because looking at my log, the proper game isn't even in the list.

[Alfagun74]

No i just checked. It says "INCLUDED_STORES":[0] in your log aswell.

I will VPN to canada and rerun the experiment to see if RAWG behaves any different overseas. This is really bugging me.

[Alfagun74]

Canada Experiment Results: gamevault-backend-maple-sirup-edition-2024-03-13.log.json

[nodiaque]

could it be because of my folder? The way I structure my stuff is I have a share with all my games. Each games have a folder where I put the supported file in it. Looking at my log, I see a lot of "ignoring unsuported extension" and this is for all of the folder it log that

edit: I just tried removing the game from the folder and same result. I'll check the unstable build. I guess I just put :unstable tag?

[Alfagun74]

I replicated your folder structure in my tests aswell so i dont see a reason for this. It's nothing unfamiliar i haven't yet seen either. The "ignoring unsuported extension" is just an informational log, and you can ignore it.

Yes the tag is gamevault-backend:unstable

[nodiaque]

samething with unstable :(

[Alfagun74]

Let me ask in our discord community if anyone experiences something similar.

[nodiaque]

In case, here's my docker-compose

[Alfagun74]

Can you Pull unstable again and grab me a fresh log again? I made them a bit more verbose.

[nodiaque]

gamevault-backend-2024-03-13.log Getting lot's of error and take 5 minutes to start with that version. Still haven't scanned the files yet edit: no it just crash edit2: reverted to latest and start, so something with the unstable build

[Alfagun74]

Apologies, my mistake. It seems I introduced a bug while under stress. I'll release a new unstable version that should be fixed shortly.

[Alfagun74]

@nodiaque could you try to pull a log with the latest unstable version?

[Alfagun74]

@nodiaque Hi, have you had any time to test this out maybe?

[nodiaque]

sorry I was away for a while. Just updated and remove the game, restart the container and no dice :(

I even have weirder things now like image that have nothing to do with anything

gamevault-backend-2024-03-29.log

[Alfagun74]

oh man, at this point i don't understand it either. You may need to wait for https://github.com/Phalcode/gamevault-backend/issues/258 to use this app properly

[Alfagun74]

Does the same issue happen when you only have a /files volume with one single game zip in it?

[nodiaque]

Yes, risk 2 is a single zip file and have this problem.

[Alfagun74]

i meant the volume should be empty with just that single file in it

[nodiaque]

Volume or folder? Risk 2 is alone in it's folder

[Alfagun74]

I want to see if it works if the entire volume mapped to /files only contains a single game zip file, nothing else.

[Alfagun74]

We'll prioritize working on task #258 with high importance. I appreciate your patience while we tackle it.

[nodiaque]

just tried with only risk 2 in a new installation. Same result, risk 2 was matched as gran turismo 2

[Alfagun74]

Im out of ideas, Sorry. Noone in the community has experienced this yet either, so i really am clueless or missing a key detail.

Oh wait!

I have one final idea. Could you hit me up on our discord or via email (contact@phalco.de) so i can send you my own rawg key in private to test if it is an issue with your API Key? Maybe Rawg just treats it different.

[Alfagun74]

Well, folks, after what feels like a coding odyssey spanning weeks and more late nights than I care to count, we've finally cracked the case! And what was the culprit, you ask? Drumroll, please... TIMEZONES?! Yes, you heard that right.

It turns out that our dear friend @nodiaque's UNRAID server possesses some mystical powers, particularly when it comes to time zones. Picture this: every container it runs defaults to "America/New York". Now, ordinarily, that shouldn't be a problem, right? We're saving our dates in the database as UTC, after all.

But hold on to your hats, because here's where it gets wild. Enter node-postgres, a seemingly innocent connection layer between Postgres and JavaScript. Innocent, that is, until you realize that it has a rather mischievous habit of converting timestamps to the local timezone before saving them into the database. Facepalm

Which Results in conversion from 2010-01-01 00:00 -> 2009-12-31 23:00 for example. Rawg searches for Games in 2009 and returns bad results. Thats the entire problem.

References:

• Node-Postgres Issue #2141
• TypeORM Issue #4519
• Node-Postgres Issue #1746
• TypeORM Issue #2390
• TypeORM Issue #2622

Fixed in v11.0.3