Scraping games for Retrobat or Batocera

[89carrot]

Hi There, thank you for building this tool! It's such a great utility.

Had a question around scraping games. I'd like to scrape from screenscraper using Skraper but noticing because the folders are a shortened version of the game name they aren't recognized. Being a folder.pc I think SS can't recognize that either.

Do you have a suggestion on how to easily get SS to recognize each of the games I convert into either Retrobat or Batocera format? It would be great to pull content such as video snaps etc for the games. Maybe even create a dummy file with the full game name.zip and then you could just edit the path of the gamelist.xml? Was also wondering if the folders still need to be shortened with Retrobat or Batocera now?

Thank you

[Voljega]

Hello !

I'm afraid there no good solution for this...

Screenscraper definitely is very picky regarding the naming (and I'm not sure their DOS scrapping is on point anyway, most likely there isn't much)

And I can't create a dummy file with a zip and edit the path of the gamelist.xml because it simply will break all the existing scrapping.

For the short folder naming and .pc, I'm not sure if it stills needed, you migh try on your side to modify a game and the gamelist with the full name of a game and tell me it works ?

If yes, it might be a solution or at least a potential way to a solution

[89carrot]

Thanks for the prompt reply!

I believe it will still work if you use the FullGameName.pc as the folder name at least in RetroBat. I renamed a game.pc folder to match the GameName.txt file in the subdirectory and then updated the gamelist.xml file to reflect this. I haven't tried in Batocera just yet.

Looking inside the folder at dosbox.bat and dosbox.cfg on a game, all the commands look to call/map/execute on the main folders subdirectory which is still the short game name and contains all the files. The .pc folder I believe all it does is get mounted in a container by dosbox and then it's contents are then mapped into a C: from the .bat/cfg files above along with the actual game files directory.

So I think if the converter is able to do the export each game to a .pc folder using the full game name, and preserve the contents inside the folder including the short game name directory, that would make it a lot better for managing the games and make it much easier to try and scrape them.

I'll test on Batocera today and report back.

Thanks!

[89carrot]

It works on Batocera also. I just copied the game.pc folder and gamelist.xml and it worked. Tried on 2 other games too.

Does your script look after creating the gamelist.xml? After scraping, the (year) is removed in the game menu - I wonder if that could be done prior when the gamelist is created?

[Voljega]

OK this is not gonna be in the coming days, but I might try to generate you a test version using the real names - might - because this is a pretty big change and I have a lot of work at my official work and pretty tired outside of it

didn't understand your last question about the year and the gamelist and the scrape ?

[89carrot]

OK this is not gonna be in the coming days, but I might try to generate you a test version using the real names - might - because this is a pretty big change and I have a lot of work at my official work and pretty tired outside of it

didn't understand your last question about the year and the gamelist and the scrape ?

I would be happy to test a version for you if you have the time. I would really appreciate it!

My last question is basically around whether or not to include the (year) portion of the game name both in the .pc folder name and also the gamelist game name. Do you think it'd be possible to have an option to choose whether to include the year in either of those names when you run an export?

Thank you for considering my request.

[Dreded]

I was just looking through the code base myself in order to make this change as right now my process in Batocera is to use this tool then run a bash script I wrote to rename all the files to the long names found in Gamelist.xml field which works well but is quit clumsy to add another game.

so if I could just have this output them as long names to begin with the problem would be solved as this tool does a good job of appending to gamelist.xml also Screenscraper has a tendency to change the names slightly(adding -'s etc)

this script is not tested extensively there is bound to be bugs but it works for me so far.

!!!!!!!!!THIS SCRIPT IS NOW USELESS ONCE THE BELOW PR IS ACCEPTED!!!!!!!!!!

!!!!!!!!!THIS SCRIPT IS NOW USELESS ONCE THE BELOW PR IS ACCEPTED!!!!!!!!!!
#!/bin/bash
for f in *.pc; do
    pathLine=$(grep -A1 "${f%.pc}.pc" gamelist.xml )

    #if we find get a result from the above grep...
    if [[ $pathLine != "" ]]; then
        #strip all before and including <name>
        name=${pathLine##*<name>}

        #strip all </name> and after
        name=${name%%<\/name>*}

        #strip all characters that are not alphanumeric or space or -(basically make a legal filename)
        name=$(echo "$name" | tr -dc '[:alnum:] -')
        name="$name.pc"

        #if we didn't already rename this game(not perfect as the scraper might have changed the game name slightly)
        #TODO: add a alreadyProcessed.csv so we can avoid some issues
        #TODO: maybe look at changing ExoDOSConverter code to add Longname checkbox to avoid all of this.
        if [[ $f != $name ]]; then
            printf "Changing: $f \tto \t$name\n"

        #replace shortname.pc with longname.pc between <path></path> in gamelist.xml and rename folder
            sed -i "s/<path>.\/$f/<path>.\/$name/" gamelist.xml
            mv "$f" "$name"

        else
            echo "Already Processed $name, Skipping"
        fi
    else
        echo "Gamelist.xml <path> doesn't contain $f"
    fi

done
echo "Done!"

!!!!!!!!!THIS SCRIPT IS NOW USELESS ONCE THE BELOW PR IS ACCEPTED!!!!!!!!!!

[89carrot]

This is great @Voljega and @Dreded thank you for working on this.

What do you think about the option for the year of the game to be included or not in the long folder name?

How do the commits work though to put together a new version on Windows? I think I just used the installer last time.

Cheers

[89carrot]

Hello @Voljega, would you know when the pull request by @Dreded will be available in a new release of the converter for Windows?

Thank you

[Voljega]

I'm in vacation for one week so I''ll try to a windows release in june

Le dim. 29 mai 2022 à 20:11, 89carrot @.***> a écrit :

Hello @Voljega https://github.com/Voljega, would you know when the pull request by @Dreded https://github.com/Dreded will be available in a new release of the converter for Windows?

Thank you

— Reply to this email directly, view it on GitHub https://github.com/Voljega/ExoDOSConverter/issues/99#issuecomment-1140489116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKXF4I47PKN7L5ESC4EZWTVMOQMFANCNFSM5OHL2VMA . You are receiving this because you were mentioned.Message ID: @.***>

[89carrot]

Sounds good and enjoy your time off!

[89carrot]

Good day @Voljega , would you have some time to create a new Windows release soon?

Thank you very much.

[Voljega]

Hello I didn't forget you ;)

Had a look again at the PR, must merge it and change a few things and then I will build you a version.

I'll try to do that this week, but it might be beginning of next week

[Voljega]

@89carrot new version has been released, please test and tell me if everything works fine

[89carrot]

@89carrot new version has been released, please test and tell me if everything works fine

@Voljega - tested and working well! Thank you for adding this feature :)

[Voljega]

thanks to @Dreded more than me :)