search

tkashkin / GameHub

All your games in one place
https://tkashkin.github.io/projects/gamehub
GNU General Public License v3.0
2.25k stars 128 forks source link

Remember which games are merged #41

Closed neuromancer closed 6 years ago

neuromancer commented 6 years ago

Every time I open GameHub, it merges all my games. This takes a considerable time with a large library (~300 games) so it will be better to cache which games should be merged and only scan for the new ones.

tkashkin commented 6 years ago

Good idea, I'll try to think how to implement it.

tkashkin commented 6 years ago

Are you sure that merging and not loading itself is bottleneck? I don't really know how to optimize merging without rewriting much of game loading code. Even if merges will be cached, iterating will still be required to exclude merged games. I don't know if caching will improve something. Maybe you can see some places to optimize which I don't see now?

neuromancer commented 6 years ago

Last revision of GameHub takes 26 seconds to start (until is ready to run some games). This is the log:

$ gamehub --debug
Registering Stacktrace 0.9.4
[INFO 16:17:27.857379] Application.vala:154: GameHub version: 0.8.0
[INFO 16:17:27.857416] Application.vala:156: Kernel version: 4.17.11-arch1
[INFO 16:17:27.983387] app.vala:32: Distro: NAME="Arch Linux"
[DEBUG 16:17:27.984757] Settings.vala:163: Loading settings from schema 'com.github.tkashkin.gamehub.ui'
[DEBUG 16:17:28.024262] Settings.vala:163: Loading settings from schema 'com.github.tkashkin.gamehub.auth.steam'
[DEBUG 16:17:28.024693] GOG.vala:136: [Auth] Refreshing GOG access token with refresh token: 
...
[DEBUG 16:17:28.742967] SoupDownloader.vala:50: [SoupDownloader] Downloading 'https://images-3.gog.com/a379891079876bb12a96a9a64e6502d967b3e76c42f13f785cd7e7965667ccdb.png'...
[DEBUG 16:17:31.498256] [GdkPixbuf] gdk_pixbuf_from_pixdata() called on:
[DEBUG 16:17:31.498294] [GdkPixbuf]     Encoding raw
[DEBUG 16:17:31.498315] [GdkPixbuf]     Dimensions: 14 x 14
[DEBUG 16:17:31.498337] [GdkPixbuf]     Rowstride: 56, Length: 808
[DEBUG 16:17:31.498358] [GdkPixbuf]     Copy pixels == false
[DEBUG 16:17:31.504477] [GdkPixbuf] gdk_pixbuf_from_pixdata() called on:
[DEBUG 16:17:31.504687] [GdkPixbuf]     Encoding raw
[DEBUG 16:17:31.504715] [GdkPixbuf]     Dimensions: 14 x 14
[DEBUG 16:17:31.504730] [GdkPixbuf]     Rowstride: 56, Length: 808
[DEBUG 16:17:31.504751] [GdkPixbuf]     Copy pixels == false
[DEBUG 16:17:32.196055] GamesView.vala:541: [Merge] Merging 'Psychonauts' (GOG:1207658807) with 'Psychonauts' (Humble Bundle:psychonauts)
[DEBUG 16:17:33.000127] GamesView.vala:541: [Merge] Merging 'Legend of Grimrock' (GOG:1207659016) with 'Legend of Grimrock' (Humble Bundle:legendofgrimrock)
[DEBUG 16:17:33.196777] GamesView.vala:541: [Merge] Merging 'Unmechanical: Extended' (GOG:1207659087) with 'Unmechanical' (Humble Bundle:unmechanical)
[DEBUG 16:17:33.302852] GamesView.vala:541: [Merge] Merging 'Deponia' (GOG:1207659103) with 'Deponia' (Humble Bundle:deponia)
[DEBUG 16:17:33.403567] GamesView.vala:541: [Merge] Merging 'La-Mulana' (GOG:1207659117) with 'La-Mulana' (Humble Bundle:la_mulana)
[DEBUG 16:17:33.479750] GamesView.vala:541: [Merge] Merging 'Hotline Miami' (GOG:1207659118) with 'Hotline Miami' (Humble Bundle:hotlinemiami)
[DEBUG 16:17:33.706932] GamesView.vala:541: [Merge] Merging 'Costume Quest' (GOG:1207659132) with 'Costume Quest' (Humble Bundle:costumequest)
[DEBUG 16:17:33.707986] GamesView.vala:541: [Merge] Merging 'Costume Quest' (GOG:1207659132) with 'Costume Quest' (Humble Bundle:costumequest_android)
[DEBUG 16:17:33.916202] GamesView.vala:541: [Merge] Merging 'Stacking' (GOG:1207659145) with 'Stacking' (Humble Bundle:stacking)
[DEBUG 16:17:34.255576] GamesView.vala:541: [Merge] Merging 'Unepic' (GOG:1207659227) with 'UnEpic' (Humble Bundle:unepic)
[DEBUG 16:17:34.267706] GamesView.vala:541: [Merge] Merging 'Amnesia: A Machine For Pigs' (GOG:1207659443) with 'Amnesia: A Machine For Pigs' (Humble Bundle:amnesia_amachineforpigs)
[DEBUG 16:17:34.526461] GamesView.vala:541: [Merge] Merging 'Democracy 3' (GOG:1207659953) with 'Democracy 3' (Humble Bundle:democracy3)
[DEBUG 16:17:34.636499] GamesView.vala:541: [Merge] Merging 'Outlast' (GOG:1207660064) with 'Outlast' (Humble Bundle:outlast)
[DEBUG 16:17:34.809700] GamesView.vala:541: [Merge] Merging 'Shadowrun Returns' (GOG:1207660413) with 'Shadowrun Returns' (Humble Bundle:shadowrunreturns)
[DEBUG 16:17:34.901458] GamesView.vala:541: [Merge] Merging 'Teslagrad' (GOG:1207660433) with 'Teslagrad' (Humble Bundle:teslagrad)
[DEBUG 16:17:34.971356] GamesView.vala:541: [Merge] Merging 'Risk of Rain' (GOG:1207660563) with 'Risk of Rain' (Humble Bundle:riskofrain_game)
[DEBUG 16:17:35.991880] GamesView.vala:541: [Merge] Merging 'Sir, You Are Being Hunted' (GOG:1207664303) with 'Sir, You Are Being Hunted' (Humble Bundle:siryouarebeinghunted)
[DEBUG 16:17:36.260932] GamesView.vala:541: [Merge] Merging 'Xenonauts' (GOG:1207664803) with 'Xenonauts' (Humble Bundle:xenonautsstrategicplanetarydefencesimulator)
[DEBUG 16:17:36.311629] GamesView.vala:541: [Merge] Merging 'OlliOlli' (GOG:1207665033) with 'OlliOlli' (Humble Bundle:olliolli)
[DEBUG 16:17:36.532804] GamesView.vala:541: [Merge] Merging 'Door Kickers' (GOG:1207666463) with 'Door Kickers' (Humble Bundle:doorkickers)
[DEBUG 16:17:36.709537] GamesView.vala:541: [Merge] Merging 'Grim Fandango Remastered' (GOG:1207667183) with 'Grim Fandango Remastered' (Humble Bundle:grimfandango)
[DEBUG 16:17:36.826612] GamesView.vala:541: [Merge] Merging 'Sunless Sea' (GOG:1421064427) with 'Sunless Sea' (Humble Bundle:sunlesssea)
[DEBUG 16:17:36.959218] GamesView.vala:541: [Merge] Merging 'Hand of Fate' (GOG:1424100574) with 'Hand of Fate' (Humble Bundle:handoffate)
[DEBUG 16:17:37.044834] GamesView.vala:541: [Merge] Merging 'Hot Tin Roof: The Cat That Wore A Fedora' (GOG:1424442317) with 'Hot Tin Roof' (Humble Bundle:hottinroof)
[DEBUG 16:17:37.129093] GamesView.vala:541: [Merge] Merging 'Hotline Miami 2: Wrong Number' (GOG:1424773427) with 'Hotline Miami 2: Wrong Number' (Humble Bundle:hotlinemiami2)
[DEBUG 16:17:37.306681] GamesView.vala:541: [Merge] Merging 'Not a Hero' (GOG:1429698467) with 'NOT A HERO' (Humble Bundle:notahero)
[DEBUG 16:17:37.518894] GamesView.vala:541: [Merge] Merging 'Breach & Clear' (GOG:1434013541) with 'Breach & Clear' (Humble Bundle:breachandclear)
[DEBUG 16:17:37.915160] GamesView.vala:541: [Merge] Merging 'Ziggurat' (GOG:1437564865) with 'Ziggurat' (Humble Bundle:ziggurat)
[DEBUG 16:17:37.946036] GamesView.vala:541: [Merge] Merging 'Frozen Synapse' (GOG:1438076423) with 'Frozen Synapse' (Humble Bundle:frozensynapse)
[DEBUG 16:17:38.061640] GamesView.vala:541: [Merge] Merging 'SOMA' (GOG:1439487606) with 'SOMA' (Humble Bundle:soma)
[DEBUG 16:17:38.187460] GamesView.vala:541: [Merge] Merging 'Dropsy' (GOG:1441869560) with 'Dropsy' (Humble Bundle:dropsy)
[DEBUG 16:17:38.297520] GamesView.vala:541: [Merge] Merging 'Prison Architect' (GOG:1441974651) with 'Prison Architect' (Humble Bundle:prisonarchitect)
[DEBUG 16:17:38.952360] GamesView.vala:541: [Merge] Merging 'Oxenfree' (GOG:1457519353) with 'Oxenfree' (Humble Bundle:oxenfree)
[DEBUG 16:17:39.023441] GamesView.vala:541: [Merge] Merging 'Jotun: Valhalla Edition' (GOG:1458127099) with 'Jotun' (Humble Bundle:jotun)
[DEBUG 16:17:39.080368] GamesView.vala:541: [Merge] Merging 'BIT.TRIP Runner' (GOG:1460734409) with 'BIT.TRIP RUNNER' (Humble Bundle:bittriprunner)
[DEBUG 16:17:39.470906] GamesView.vala:541: [Merge] Merging 'Torchlight II' (GOG:1958228073) with 'Torchlight II' (Humble Bundle:torchlight2)
[DEBUG 16:17:39.632990] GamesView.vala:541: [Merge] Merging 'THE KING OF FIGHTERS 2002' (GOG:2114035213) with 'THE KING OF FIGHTERS 2002' (Humble Bundle:thekingoffighters_2002_base)
[DEBUG 16:17:46.920822] [GdkPixbuf] gdk_pixbuf_from_pixdata() called on:
[DEBUG 16:17:46.920902] [GdkPixbuf]     Encoding raw
[DEBUG 16:17:46.920936] [GdkPixbuf]     Dimensions: 14 x 14
[DEBUG 16:17:46.920956] [GdkPixbuf]     Rowstride: 56, Length: 808
[DEBUG 16:17:46.920979] [GdkPixbuf]     Copy pixels == false
[FATAL 16:17:46.994935] [Json] json_node_get_array: assertion 'JSON_NODE_IS_VALID (node)' failed
An error occured (SIGTRAP)
   The reason is likely an uncaught error.
   Note: no file path and line numbers can be retrieved. Are you sure you added -g -X -rdynamic to valac command line?

    #1  <unknown>   in 'json_node_get_array'
        at /usr/lib/libjson-glib-1.0.so.0
    #2  <unknown>   in 'g_main_context_dispatch'
        at /usr/lib/libglib-2.0.so.0
    #3  <unknown>   in 'g_main_context_iteration'
        at /usr/lib/libglib-2.0.so.0
    #4  <unknown>   in 'g_application_run'
        at /usr/lib/libgio-2.0.so.0
    #5  <unknown>   in '__libc_start_main'
        at 
[FATAL 16:17:47.581176] [Json] json_array_get_elements: assertion 'array != NULL' failed
An error occured (SIGTRAP)
   The reason is likely an uncaught error.
   Note: no file path and line numbers can be retrieved. Are you sure you added -g -X -rdynamic to valac command line?

    #1  <unknown>   in 'json_array_get_elements'
        at /usr/lib/libjson-glib-1.0.so.0
    #2  <unknown>   in 'g_main_context_dispatch'
        at /usr/lib/libglib-2.0.so.0
    #3  <unknown>   in 'g_main_context_iteration'
        at /usr/lib/libglib-2.0.so.0
    #4  <unknown>   in 'g_application_run'
        at /usr/lib/libgio-2.0.so.0
    #5  <unknown>   in '__libc_start_main'
        at 
[DEBUG 16:17:48.196299] SoupDownloader.vala:127: [SoupDownloader] Content-Length: 29165
[DEBUG 16:17:48.196433] SoupDownloader.vala:156: [SoupDownloader] '/home/g/.cache/com.github.tkashkin.gamehub/images/a379891079876bb12a96a9a64e6502d967b3e76c42f13f785cd7e7965667ccdb.png' exists
[DEBUG 16:17:48.197066] SoupDownloader.vala:80: [SoupDownloader] Downloaded 'https://images-3.gog.com/a379891079876bb12a96a9a64e6502d967b3e76c42f13f785cd7e7965667ccdb.png'
[DEBUG 16:17:48.523500] GOG.vala:193: [GOG] Loading games: page 1 of 3
[DEBUG 16:17:49.445073] GOG.vala:193: [GOG] Loading games: page 2 of 3
[DEBUG 16:17:50.155722] GOG.vala:193: [GOG] Loading games: page 3 of 3
tkashkin commented 6 years ago

Will it launch faster with merging disabled?

neuromancer commented 6 years ago

It takes around ~7 seconds to run without merging.

tkashkin commented 6 years ago

Ok, I have an idea how to implement it but I'm not sure how much will it help.

tkashkin commented 6 years ago

Also what's with json parsing errors? Can you run it with gdb and get a backtrace? gdb gamehub, r, bt when it hits breakpoint, c to continue.

neuromancer commented 6 years ago 
[FATAL 16:39:30.886980] [Json] json_node_get_array: assertion 'JSON_NODE_IS_VALID (node)' failed

Thread 1 "gamehub" received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffff75c0ed2 in ?? () from /usr/lib/libglib-2.0.so.0
(gdb) bt
#0  0x00007ffff75c0ed2 in  () at /usr/lib/libglib-2.0.so.0
#1  0x00007ffff75c223d in g_logv () at /usr/lib/libglib-2.0.so.0
#2  0x00007ffff75c23a0 in g_log () at /usr/lib/libglib-2.0.so.0
#3  0x00007ffff2eac55b in json_node_get_array () at /usr/lib/libjson-glib-1.0.so.0
#4  0x00005555555713f1 in  ()
#5  0x0000555555591585 in  ()
#6  0x00007ffff6110204 in  () at /usr/lib/libgio-2.0.so.0
#7  0x00007ffff6110d07 in  () at /usr/lib/libgio-2.0.so.0
#8  0x00005555555921a1 in  ()
#9  0x0000555555591525 in  ()
#10 0x00007ffff6110204 in  () at /usr/lib/libgio-2.0.so.0
#11 0x00007ffff6110d07 in  () at /usr/lib/libgio-2.0.so.0
#12 0x000055555559190e in  ()
#13 0x00007ffff75bb1d6 in g_main_context_dispatch () at /usr/lib/libglib-2.0.so.0
#14 0x00007ffff75bb5b1 in  () at /usr/lib/libglib-2.0.so.0
#15 0x00007ffff75bb63e in g_main_context_iteration () at /usr/lib/libglib-2.0.so.0
#16 0x00007ffff612697e in g_application_run () at /usr/lib/libgio-2.0.so.0
#17 0x000055555556019b in  ()
#18 0x00007ffff25ce06b in __libc_start_main () at /usr/lib/libc.so.6
#19 0x000055555555fdba in  ()
(gdb) info registers 
rax            0x0  0
rbx            0x1  1
rcx            0x5555579be680   93825030416000
rdx            0x0  0
rsi            0x5555579be680   93825030416000
rdi            0x1  1
rbp            0x8  0x8
rsp            0x7fffffffe320   0x7fffffffe320
r8             0x555556838da0   93825012043168
r9             0x7  7
r10            0x0  0
r11            0x0  0
r12            0x8  8
r13            0x8  8
r14            0xa  10
r15            0x3  3
rip            0x7ffff75c0ed2   0x7ffff75c0ed2
eflags         0x202    [ IF ]
cs             0x33 51
ss             0x2b 43
ds             0x0  0
es             0x0  0
fs             0x0  0
gs             0x0  0
(gdb) x/i $rip
=> 0x7ffff75c0ed2:  pop    %rbx
tkashkin commented 6 years ago

Not too much help with debug symbols stripped unfortunately. Try to rebuild with --buildtype=debug passed to meson or meson configure.

tkashkin commented 6 years ago

Try b02381e. Games loading and merging are now multithreaded. App should load and be usable much faster, but may become less stable. Also check if #31 is fixed. There may be some false-positive merges, but it should work fine most of the times.

Due to threading there's a new bug in merging related to game load order. I'll try to fix it now. image

neuromancer commented 6 years ago

Try b02381e. Games loading and merging are now multithreaded. App should load and be usable much faster, but may become less stable.

The app seems to be loading faster, but I cannot click in any games or review the list of them until most of them are load. Anyway, it is definitely an improvement from the previous version.

tkashkin commented 6 years ago

Try ee79571. Merging is improved. Now it correctly handles games with more than 2 sources. It also merges games in priority. It may also be a little faster.

31 should be fixed, now it will display one game per source.

I've also noticed that games counter in titlebar is now invalid if there are merged games. Do you think it's useful or should it be removed?

neuromancer commented 6 years ago

It may also be a little faster.

It's faster. It let me navigate my list of games after 8 or 10 seconds. If you think it's the best you can do without re-writting everything, feel free to close this issue.

I've also noticed that games counter in titlebar is now invalid if there are merged games. Do you think it's useful or should it be removed?

IMHO, this is a very minor issue. I think it's useful to have an (under) approximation of the number of games.

tkashkin commented 6 years ago

Yes, I don't think it can be optimized more easy. Closing.

Feel free to reopen if merging false-positives or if something breaks in merging or threading.

Btw game counter is fixed.