Idea to incorporate sselph/scraper in EmulationStation

[sselph]

I had an idea that could help reduce the extra code and maintenance for something that used my code in the existing ES UI. I could write a simple web server that receives the rom path and returns the data that is processed by ES. Similar to https://github.com/Aloshi/EmulationStation/commit/2bd9062c2dcde0c7d2105d48c0c6249ca168cf30 but the hashing takes place in my Go code so I can maintain the bulk of the logic in my repo. It is more code and doesn't work as well for multi file formats(bin/cue). It is also slower than my plain script since the ES code is has to update UI elements, but it might be a better UX for people that don't read the wiki.

I opened this just for discussion. If it is something you wanted to pursue, we could discuss more in detail about the approach. I don't think it would take much on my side to spin something up since Go has standard libraries for creating a web server.

[HerbFargus]

I personally think its a great idea seeing as your scraper is lightyears ahead of the built in scraper especially with all the recent no-intro updates.

I'm a terrible coder but I am a good tester ;)

[sselph]

I don't read c++ well so that would probably be the most time consuming part. Luckily retropie only runs on linux so I don't have to worry about figuring out if the build works in windows. The only concern I have is if the scraper returns no results, I think the current logic is to prompt the user unless an option is set to not do this. We'd probably want to set this option always for my scraper or have a trigger from the scraper that a match isn't possible and skip regardless of the setting. I'll try to review the ES code again.

[sselph]

I was able to write a proof of concept for this. It is a small web server that runs and has an api exposed /getgame?path= and that returns the data from thegamesdb. Then wrote a simple scraper module for ES that passes the path to the api and then parses the result.

Only issue is even if I return 1 result ES pauses like there are more than one and is asking me to choose like there is a conflict. So you have to turn off the user decides on conflict. I'll continue to play around with it.

[sselph]

Actually, I just noticed there is already a provision for automatically accepting a result if there is a CRC match. I'd just need to signal the match. I also have code already written to do the types of searches that ES does so could potentially try and match the hash, and if it fails fallback to the standard behavior.

[sselph]

Okay I have this in a working state. I implemented 2 scrapers one that is similar to what I do today and just scrapes everything without prompting and one with a manual drop out when things aren't automatically scraped and that mimics the current scraping logic.

What is the best way to proceed with some testing? I can check in the ES code to my fork, send a PR, send a patch. etc. I can check in the code for the scraper web service or post a binary.

[HerbFargus]

For me if you submit a pull request I'm happy to give it a test, I'm not sure what @joolswills prefers or what his thoughts are on the matter

[sselph]

Yeah I guess @joolswills 's opinion is important. My main concern is that it does complicate the system by requiring a second binary to be running. So installing, updating, and making sure it is running. But it does have the benefit of offloading most of the scraping logic.

Tomorrow evening I'll send a PR to this repo and commit my service with some instructions to test it.

[sselph]

Created https://github.com/RetroPie/EmulationStation/pull/51

I uploaded versions of the scraper utility here: https://storage.googleapis.com/stevenselph.appspot.com/web_linux_386.zip https://storage.googleapis.com/stevenselph.appspot.com/web_linux_amd64.zip https://storage.googleapis.com/stevenselph.appspot.com/web_rpi.zip https://storage.googleapis.com/stevenselph.appspot.com/web_rpi2.zip https://storage.googleapis.com/stevenselph.appspot.com/web_windows_386.zip https://storage.googleapis.com/stevenselph.appspot.com/web_windows_amd64.zip https://storage.googleapis.com/stevenselph.appspot.com/web_mac_386.zip https://storage.googleapis.com/stevenselph.appspot.com/web_mac_amd64.zip

So you'd make it work by running the web application in the zip and the updated ES. Then choose sselph or sselph w/fallback. The plain sselph is the one that just runs and the w/fallback will run until it hits something it can't scrape then show the search results with the option to update the search like the current scraper does it.

[sselph]

So I have been revisiting this and have been able to compile my code as a C library that I imported it into ES and was able to use my code directly from ES. The UI could use some improvements, but it works. Now it comes down to what is the easiest way to update the build to allow for it.

From my side I will create a new project or folder that is the library with a make file. It can compile and install the library on linux. This will require Go1.5+ which isn't the default in wheezy. It is super easy to install but it is another step. So I could also provide pre-compiled libraries via github, etc. I would also update ES's CMake to find the installed library.

There are a couple issues: Windows - I know RetroPie isn't supported on windows but I know people use it for development. I have no way of testing builds on it. I will see if my ancient XP CD still works and experiment. I know for a fact I can't build a .dll but I might be able to build a static library. Another option is I could create a dummy library that has the same API but does nothing that can be included in ES and compiles when my library isn't there. Another option is to figure out how to dynamically load the library(dlopen/LoadLibrary) so even if it is missing ES works but has less options. This would be ideal but getting to the limits of my C++ knowledge.

If I provide pre-compiled libraries, I need to know the architectures and be able to compile for them. At the moment the only things I have access to are linux/arm(retropie image) and freebsd/amd64 so I could definitely do the standard retropie on a raspberry pi libraries but others may have to compile from source or we do the dynamic thing mentioned above.

[HerbFargus]

by default in wheezy did you mean jessie? retropie builds on raspbian Jessie, we actually no longer even support wheezy.

Ideally I would think we would want to be able to build it as part of the emulationstation module so you wouldnt have to create precompiled libraries and there are functions for specifying platforms (pi vs x86 etc) Though I presume there may be some systems that use emulationstation that aren't using retropie or if people just choose to use it standalone that might benefit from precompiled libraries- though I don't personally know of anybody.

As far as windows goes, I personally don't really use it all the often though I know there are some in the retropie community that have been doing tests with emulationstation on windows.

As always I'm happy to test implementations

[sselph]

Oops yeah it isn't the default in either :) wheezy used 1.2.1 and jessie uses 1.3.3 I think. The install is: wget https://storage.googleapis.com/golang/go1.7.4.linux-armv6l.tar.gz sudo tar -C /usr/local -xzf go1.7.4.linux-armv6l.tar.gz if path doesn't contain /usr/local/go/bin add it

So it is fairly straightforward. I could probably write some bash that could detect and remove the apt-get version then install the modern version. Once that is in place the build is trivial. So it could probably be hooked in to the dependency area of the emulationstation.sh file.

[HerbFargus]

yeah the go version can be easily sorted as a dependency so that should be the least of our troubles.

[sselph]

Sounds good. I just finished a large change to my code to better handle ScreenScraper. Once I get the library code working with that, I'll check it in. Then will follow up with a fork of EmulationStation and RetroPie-Setup that can handle building the new code.

[HerbFargus]

I'll be happy to test when it's ready. Primary thing is to get it more functional than the current es scraper, then perhaps it may be worth looking at possible implementations of some of the advanced settings your scraper provides

[sselph]

My idea, if you saw my code in the post on the forum is to add 3 auto scrapers that function similar to my script with the defaults. I'll also, in the same update or possibly a followup, update the current builtin to hash then, if that fails, fallback to a name search like it is currently doing. The options on naming, image type, etc will all be left out. If they are advanced enough to know they want more options they will probably figure out using the script. But it would be nice to continue on the path of improving the native UI if possible.

[sselph]

Okay I think I have something workable. It isn't great since I'm sure there are better ways to handle the installs of binaries.

https://github.com/sselph/RetroPie-Setup

If you pull that then look under the exp you will see a few packages but one is emulationstation-sselph and that pulls my new version of EmulationStation with a dependency on libscraper. I ran it successfully but had issues in that the resulting binary was printing debug log info that isn't checked in. So I'm not sure if my environment is messed up.

[HerbFargus]

It's a great first pass. I'm giving compilation a go now

[HerbFargus]

Heres the build log with errors for reference, looking at it a bit more:

http://pastebin.com/izX9YiqQ

[sselph]

Thanks. Looks like Go isn't being installed correctly. I'll see if I can figure this out with a fresh install. It is supposed to remove the previously installed Go but I'm also referencing the installed Go with the full path so could potentially ignore the apt-get install versionl.

[sselph]

Oh this just might me not doing the install correctly. You see the Installing 'golang' : never appears. If you were to go to that menu entry and install it then it would probably work. I'll update the script.

[HerbFargus]

I am trying to get a log of the text that flashes when I run sudo ./retropie_setup which usually indicates something its not happy with with a module. but yeah it seems to be a hangup with the go install at some point ( the whole setup script modules make things simpler but at the same time sure can complicate things)

[HerbFargus]

Here it is:

/home/pi/RetroPie-Setup/scriptmodules/supplementary/golang.sh: line 32: syntax error in conditional expression
/home/pi/RetroPie-Setup/scriptmodules/supplementary/golang.sh: line 32: syntax error near `"go1.7.5"'
/home/pi/RetroPie-Setup/scriptmodules/supplementary/golang.sh: line 32: `    if [[ "${version}" >= "go1.7.5" ]]; then'

[sselph]

good catch on that error I missed it. Bash is not my forte. I converted >= to > and I don't see the error.

[HerbFargus]

can confirm that fixed it. its now compiling

[sselph]

Cool. Get some popcorn cause it takes 30m or so. I used -v when possible to give you something to look at :)

[HerbFargus]

eh. spoke too soon. new build log errors:

http://pastebin.com/sdBgHLXR

still seems to be hanging up on the go install

[sselph]

well shit was about to say that:

Found VLC: /usr/include/vlc
-- Found scraper: /usr/lib/libscraper.so
-- Found ALSA: /usr/lib/arm-linux-gnueabihf/libasound.so (found version "1.0.28")
-- Configuring done

is the important part.

Seems

mv: cannot stat ‘/home/pi/RetroPie-Setup/tmp/build/libscraper/libscraper.h’: No such file or directory
mv: cannot stat ‘/home/pi/RetroPie-Setup/tmp/build/libscraper/libscraper.so’: No such file or directory
/home/pi/RetroPie-Setup/tmp/build/emulationstation-sselph /home/pi/RetroPie-Setup

Is telling

[sselph]

I assume usr/local/go/bin/go version returns No such file or directory ??

[HerbFargus]

correct.

[HerbFargus]

from the log:

Downloading go1.7.5.linux-armv6l.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Failed to create the file
Warning: /home/pi/RetroPie-Setup/tmp/build/golang/golang.tar.gz: No such file 

either didn't download properly or something else? perhaps I'm looking at it wrong. still looking at it

[HerbFargus]

btw I'm testing on my RPI3 for reference

[sselph]

That seems like it could be it. It wasn't able to save to $md_build I switched to /tmp whic might temporarily fix the issue.

I'm happy. I couldn't figure out why this worked. Turns out it doesn't :)

[HerbFargus]

I changed the last two $md_build to $md_inst instead and seems to be building now (crosses fingers)

Anyways I'm sure @joolswills will have some suggestions on how to optimise the modules. I'm just excited to get it running

[sselph]

Ah that could be it. I don't know all the directories and functions that are involved. I am basically just downloading a tar.gz file and putting it in /usr/local whatever works :)

[HerbFargus]

Just FYI he's done some API docs here: https://retropie.org.uk/retropie-setup-api/

[sselph]

thinking about it more, you are probably right. I'm only implementing install so the md_install seems reasonable. I'm aware the code is not great I'm sure @joolswills will have a laugh at my bash :) Hopefully you can get it running tonight. It is kinda neat.

[HerbFargus]

I'm running short on time, I think my SD card has finally croaked, its been stuck at 23% for a while, so I may have to revisit it later with a diff SD, (hopefully tomorrow pending work and other life annoyances) anyways thanks, good work thus far, I'm excited for this to be included into ES.

[sselph]

:) yeah It has taken this long because I was too lazy to figure out how to cache the ES build. Every attempt takes 30m to 60m.

Thanks for all the help. I'll reflash my rpi tomorrow to get some better troubleshooting.

[HerbFargus]

note to self. dont compile with kodi running. got it running, works great! much faster. may need a few tweaks but very happy with the results

[sselph]

cool if there is something I can do let me know.

[sselph]

I tested this last night with a fresh install and it seemed to work okay. There were some concerns. The CPU which was initially hovering around 30-50% in testing is now sitting at 100% the entire time. I don't know what is causing it but will work on hunting that down. Also ScreenScraper is frequently overloaded and the API locksdown to registered users. Since we have no UI to enter credentials for the site it is a less than ideal UX.

[HerbFargus]

I think there will always be a bottleneck with any API we use considering the size of our user base. As an alternate I think libretro is onto something with their snaps repositories (primarily for those that use full sets) I think I could also be more clear in the documentation about backing up scrapes as well as I bet many people just rescrape any time they reimage. That was part of my argument for including scrapes in rom folders rather than the emulationstation folders.

[joolswills]

I apologise for not having yet commented on the new work. Thanks very much - been very busy on other retropie related things. I plan to look at this in the next couple of days

PS. I sent a small donation to your email address @sselph but perhaps I had the wrong address - please can you email me (buzz [at] exotica.org.uk)

Thanks very much for your work on this.

[einsteinx2]

Any update on this? I know scraper ships as an option to install in the RetroPie, but it functions as a standalone tool. It seems like this was mostly completed. I've been using scraper on my desktop and then moving files over, as I find it faster that way, but it would be nice to have it in the UI.

[melroy89]

The default scraper currently installed within EmulationStation is the built-in one right (not the one from sselph/scraper )?

What are the benefits of the sselph scraper vs the current one? And should I request an API key for the new thegamesdb API v2..? I'm contacting Zer0xFF via discord.

[melroy89]

@joolswills Please, request an API key for v2 via the forum: https://forums.thegamesdb.net/viewforum.php?f=10

The main repo maintainer normally maintain the key. Sselph already requested one for himself, but maybe its a good idea to have one for emulationstation as well (since we do have two different scrapers here).

[einsteinx2]

The default scraper currently installed within EmulationStation is the built-in one right (not the one from sselph/scraper )?

Yes that's correct. To use sselph's scraper you have to drop into the terminal.

What are the benefits of the sselph scraper vs the current one?

I don't have an exhaustive list but in general it's faster and more accurate, or at least that's been my experience. And especially on my computer using multiple threads it's super fast (no reason that shouldn't work in RetroPie also, though the included version right now seems to be single threaded, or at least I remember it being so. In any case it's much slower than doing it on my laptop).