[Suggestion] RetroPie Module

[HerbFargus]

Perhaps I should open up this ticket on the retropie github page rather than here, but as your scraper seems to be quite a bit more effective than the built in scraper for emulationstation- Do you think its viable to code your scraper as a retropie setup-script module? Granted it would probably run from the setup script rather than from emulationstation like the other one, but I think it would still work that way.

[sselph]

I proposed this several months ago in https://github.com/RetroPie/RetroPie-Setup/issues/578 But we concluded it would be much better as part of emulation station so I checked in https://github.com/Aloshi/EmulationStation/pull/355

Since aloshi got a job development has slowed down, so this hasn't made it to the stable branch. I've also been busy but if someone had time I don't mind someone adding in a module to run this. I'm happy to provide advice.

I wrote a bash based update check for another script here that does something similar to what you probably want https://github.com/Death259/PiAssist/pull/18

[HerbFargus]

Yeah emulationstation development has kinda been at a standstill for a while so I figure it's best to just integrate stuff with retropie while I can as there's no telling when emulationstation will resume development, and if/when it does resume in the future then our changes can be adapted and pushed upstream. Personally I think your scraper should be the default as it runs much faster and is more accurate.

I like what you've done with the piassist script, I've sorta been using that as a base for adapting into a retropie module. One thing that I'm curious about is the emulationstation processes- I know that you say they should be killed before you do any scraping- but it's hard to script a kill process when it's set up to automatically reboot within 5 seconds anytime the process is killed. Since the RetroPie setup script is in the retropie menu in emulationstation- it would automatically imply emulationstation will be running when a lot of the people run the script- is there a way to automate it so that people don't have to exit all the way out and restart the setup script from the terminal, or will they have to start the setup script from the terminal if they wish to use the scraper?

Just as a note- I tested it by opening up the setup script from emulationstation and running the script to scrape all roms and upon return to emulationstation, all the artwork was there and it didn't seem to have any issues- so I guess I'm just not sure where the issue lies in scraping while emulationstation is running.

Just in case you're interested, this is where I'm at so far- all I'm missing is the checklist to select by system, everything else seems to be working so far. I named it scraper.sh and placed it in /home/pi/RetroPie-Setup/scriptmodules/supplementary/scraper.sh and then I could access it from the setup script in option 3- I titled it Stephen Selph's Scraper. I also added two little lines in the menu heading showing the latest online version and the current version installed on the pi- so people will know when to update.

#!/usr/bin/env bash

# This file is part of RetroPie.
# 
# (c) Copyright 2012-2015  Florian Müller (contact@petrockblock.com)
# 
# See the LICENSE.md file at the top-level directory of this distribution and 
# at https://raw.githubusercontent.com/RetroPie/RetroPie-Setup/master/LICENSE.md
#

rp_module_id="scraper"
rp_module_desc="Steven Selph's Scraper"
rp_module_menus="3+"
rp_module_flags="nobin"

function scraper_install() {
echo "Downloading and Installing Scraper created by SSELPH..."
scraperVersion=$(wget -qO- https://api.github.com/repos/sselph/scraper/releases/latest | grep tag_name | sed -e 's/.*"tag_name": "\(.*\)".*/\1/')
    if isPlatform "rpi2"; then
        wget https://github.com/sselph/scraper/releases/download/$scraperVersion/scraper_rpi2.zip -q
        unzip scraper_rpi2.zip scraper -d /usr/local/bin/
        rm scraper_rpi2.zip
    else
        wget https://github.com/sselph/scraper/releases/download/$scraperVersion/scraper_rpi.zip -q
        unzip scraper_rpi.zip scraper -d /usr/local/bin/
        rm scraper_rpi.zip
    fi
    printMsgs "dialog" "$scraperVersion Scraper Installed"
}

function scraper_all() {
su pi -c "/usr/local/bin/scraper -scrape_all -thumb_only"
printMsgs "dialog" "All games scraped. You can now restart EmulationStation by typing emulationstation into the terminal."
}

function scraper_system() {
true;
}

function scraper_emulationstation() {
printMsgs "dialog" "Killing emulationstation- restart the setup script from the terminal by typing:\ncd RetroPie-Setup && sudo ./retropie_setup.sh"
exec killall emulationstation
}

function configure_scraper() {
    scraperVersion=$(wget -qO- https://api.github.com/repos/sselph/scraper/releases/latest | grep tag_name | sed -e 's/.*"tag_name": "\(.*\)".*/\1/')
    currentVersion=$(/usr/local/bin/scraper --version)

    while true; do
        cmd=(dialog --title "Steven Sselph's Scraper."  --backtitle "$__backtitle" --menu "Latest Version:    $scraperVersion\nInstalled Version: $currentVersion" 22 76 16)
        options=(
            1 "Install Latest Scraper"
            2 "Scrape All Systems"
            3 "Scrape System(s) From List"
            4 "Kill EmulationStation"
        )
        choice=$("${cmd[@]}" "${options[@]}" 2>&1 >/dev/tty)
        if [[ -n "$choice" ]]; then
            case $choice in
                1)
                    scraper_install
                    ;;
                2)
                    scraper_all
                    ;;
                3)
                    scraper_system
                    ;;
                4)
                    scraper_emulationstation
                    ;;
            esac
        else
            break
        fi
    done
}

[sselph]

Very nice. I've never had issues and never really did any tests to see if there were issues. If there are issues, I think it might be that ES waits until it is closed to write to disk and it might do that regardless of things being edited inside ES. I think if you trigger the script from ES it might solve that issue.

[sselph]

Also, right now -scrape_all doesn't know to change settings when it finds an arcade platform. I'll fix that soon.

[HerbFargus]

That will be a very useful change- I'm looking forward to it. Looks like jools will be integrating it into the retropie setup script sometime in the future, so I'll close this issue. Thanks for the input and your work with this scraper :)