Thumbnail downloader issues

andres-asm commented 8 years ago

Had a conversation last night with leiradel and we concluded the current approach won't scale. The conversation: http://hastebin.com/odocajumog.xml

So, monolithic download pros:

single download, should be faster if the user wants all the thumbs since it would mean less requests
virtually zero infrastructure changes
once unzipped, it's done, just works

monolithic download cons:

huge file, should take a while to download and extract, many users will hammer the download every week or so even if there are no changes, wasting our bandwidth and killing their storage on the way

On-demand download pros:

small downloads, should take just a few seconds per-thumbnail on a decent connection
acts like a cache, once a file is fetched it wouldn't be downloaded
again no infraestructure changes
if a file is added it could be updated without downloading 300MB again
can be cached by a CDN making the bandwidth requirement on server side virtually zero
the approach could scale to download cheats, icons, saves, patches, anything we want

On-demand download cons:

noticeable delay before a thumbnail appears (the first time), same thing happens virtually anywhere, heck even loading the thumbnails from the cache on steam takes a couple seconds
more code needed on the RetroArch side
needs a few alterations to the OSD interface to avoid spamming messages

Anyway, my $0.02, it would be up to you guys, I just thought I should make a list and try to weigh pros and cons.

@Kivutar @leiradel @twinaphex

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/34335268-thumbnail-downloader-issues?utm_campaign=plugin&utm_content=tracker%2F296058&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F296058&utm_medium=issues&utm_source=github).

kivutar commented 8 years ago

Hi, the main con of the on-demand way is not listed: it is slow. It takes 5 hours to download the whole SNES set (versus 2 minutes with the pack)

Alcaro commented 8 years ago

Maybe we could use monolithic for the first download then update via on-demand?

It would double the server disk space use, but it should also combine most of the advantages of the two methods.

kivutar commented 8 years ago

It could work, but maintaining both codepath have a price too.

Are there any ways to make the on demand downloads faster?

Another downside that have not been listed for the on-demand way: while we spawn all these http requests, displaying a stored image takes more time, the delay is noticeable and it's not nice.

leiradel commented 8 years ago

An idea that appears in the IRC conversation linked above is to have a server-side script that would take a list of files to update and return them packaged i.e. in a tar (because the tar can be built on-the-fly without requiring a temporary file on the filesystem.) Maybe we could use this in place of the huge zip download?

Pros: only downloads images for games the user has in the playlists (less bandwidth and download time) Cons: content is not cacheable (but neither are zips)

As for making on-demand downloads faster, there's not much to be done I think. I've implemented a way to cancel HTTP tasks, so if the user browsed to another game before the thumbnail was downloaded, the task was cancelled and then another download was started. Maybe you didn't try this version @Kivutar ?

andres-asm commented 8 years ago

The SNES set took 5 minutes on my connection. There isn't any technical reason for it being slow besides users being in a slow or high latency connection.

The only difference (if the server is the same) would be the time to establish one TCP connection per-file. It should definitely not amount to more than a sec per-file.

On Wed, May 18, 2016, 12:27 PM Andre Leiradella notifications@github.com wrote:

An idea that appears in the IRC conversation linked above is to have a server-side script that would take a list of files to update and return them packaged i.e. in a tar (because the tar can be built on-the-fly without requiring a temporary file on the filesystem.) Maybe we could use this in place of the huge zip download?

Pros: only downloads images for games the user has in the playlists (less bandwidth and download time) Cons: content is not cacheable (but neither are zips)

As for making on-demand downloads faster, there's not much to be done I think. I've implemented a way to cancel HTTP tasks, so if the user browsed to another game before the thumbnail was downloaded, the task was cancelled and then another download was started. Maybe you didn't try this version @Kivutar https://github.com/Kivutar ?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/libretro/RetroArch/issues/3010#issuecomment-220099514

leiradel commented 8 years ago

I've converted the SNES thumbnails, and while the size is smaller for the boxarts, it's bigger for the other two sets:

Boxarts: png=232 mb vs jpeg=86,1 mb Snaps: png=29,7 mb vs jpeg=137 mb Titles: png=29 mb vs jpeg=123 mb

So I think it's worth the trouble converting but only for the package downloader, the on-demand downloader will get very little benefits from the conversion wrt delays (taking into account that the delay for png snaps and titles are considered unacceptable.)

RobLoach commented 8 years ago

PNG 232+29+29 = 290 JPG 86+137+123 = 346

In general, the Boxart is of a pre-rendered graphic, and a larger image size, so JPG makes sense for the Boxart. Could we support loading either JPG or PNG in Retroarch, and then switch the Boxarts to JPG? If we did that, the result would be:

86+29+29 = 144

leiradel commented 8 years ago

RA already supports both formats so I guess we'll end up converting the boxart to jpeg, but that alone won't solve the problems we're trying to solve.

kivutar commented 8 years ago

I will try again as soon as the server is fixed. But I believe that I tested the latest code.

Generating the pack on the server side could be done yeah.

If the code improved enough to get the whole snes set in 5 min, then I'm all for it!

andres-asm commented 8 years ago

Can we close this now @Kivutar @RobLoach ?

kivutar commented 8 years ago

I'm not sure if that conversation is over yet.

lasers commented 7 years ago

I think the conversation link is dead. There are 73 directories right now on https://github.com/libretro/libretro-thumbnails and not everything will be named accordingly right away when we put them up on Github.

This mean the users are already downloading bunch of files for no reasons. If we want to, my computer and your smartphone can download Sony - PlayStation3.zip or Sony - PlayStation4.zip for lulz or by accident.

Thumbnails Updater could do some simple file check for cores and playlists in the background first before giving users a list to download. This way, we could display only the systems with installed cores and compiled playlists by default.

For first timers, there will be nothing to see under that option. The users will have to scan their contents first. The Thumbnails Updater then will display Famicon, NES, and SNES until they scan some new contents for different systems first.

Cut down on unnecessary content downloading. This could apply to just about anything including cheats. Some users will have no need for Genesis cheats or Gameboy cheats especially if they don't have Genesis or Gamebvoy cores (and/or playslists) in first place... So why bother downloading them?

760K    ./Sega - Saturn
2.8M    ./Nintendo - Nintendo Entertainment System
12K ./Computer - Games
1.3M    ./Sega - Master System - Mark III
5.0M    ./Nintendo - Super Nintendo Entertainment System
8.2M    ./Sony - PlayStation
2.8M    ./Nintendo - Game Boy
2.0M    ./Nintendo - Game Boy Advance
2.0M    ./Nintendo - Game Boy Color
28K ./Nintendo - Satellaview
3.1M    ./Sega - Mega Drive - Genesis
1.6M    ./Sega - Game Gear
8.0K    ./Sega - Mega-CD - Sega CD
30M .

The cheat example might be a little too extreme, but the idea is to let users download what they can right now and not putting everything out in the open. PS Vita users does not need Sony - PlayStation2.zip either if it's not capable of playing one.

If RetroArch team decides to add new banners, short clips, or whatever it is in the future, this checking system would still apply... and still would save bandwidth. See how Core Updater and Content Downloader do things right now.

lasers commented 7 years ago

This is not related to Thumbnails Updater but still falls under Online Updater.

I suggest to split up Update Assets... into Update Assets and Update Wallpapers

Everytime I Update Assets, I'm downloading wallpapers. I already looked at them and know I don't want to use them anyway... so I think people may be updating assets regularly and not giving any crap about the wallpapers.

792K  ./pkg         792K  ./pkg
24M   ./wallpapers  26M   ./xmb
26M   ./xmb         100K  ./nuklear
100K  ./nuklear     1.9M  ./glui
1.9M  ./glui        7.0M  ./custom
7.0M  ./custom      11M   ./branding
11M   ./branding    176K  ./zarch
176K  ./zarch       46M   .
70M   .

Less than almost half.

EDIT: Almost all src in ./xmb got removed. New number.

49M .

Removing wallpapers would cut it down to...

25M

RobLoach commented 4 years ago

The new playlist thumbnail downloader grabs just thumbs from your playlist :+1:

No more legacy pack downloader.

libretro / RetroArch

Thumbnail downloader issues #3010