search

muldjord / skyscraper

Powerful and versatile game scraper written in c++
GNU General Public License v3.0
476 stars 127 forks source link

cache report:missing CLI option honour --nosubdirs #163

Closed vanativo closed 5 years ago

vanativo commented 5 years ago

Describe the feature Currently when using the --cache report:missing command line option it reports missing resources for ALL roms in the path including sub-directories. This means the output includes many files which I don't care about since I originally gathered the resources using that option, so the only files I care about are at the top level. This makes it more difficult to isolate the real problems / missing resources.

Contact mails and documentation NA

Additional context Example, I have my amiga roms organized into one directory per game in order to hold multi-disk games. I then create a symbolic link to the first disk of each game at the parent level. Eg:

drwxr-xr-x 2 pi pi 4096 Aug 22 20:40 Dragon's Lair lrwxrwxrwx 1 pi pi 109 Aug 22 22:04 Dragon's Lair.adf -> /home/pi/RetroPie/roms/amiga/Dragon's Lair/Dragon's Lair (1987)(ReadySoft)[cr ACU][t +2 ACU](Disk 1 of 8).adf

When I then run the following command: /opt/retropie/supplementary/skyscraper/Skyscraper -p amiga -s screenscraper -u XXXX:XXXX --videos --nosubdirs --refresh

I only get matches attempted on the symbolic links which is what I want.

If I then run:

/opt/retropie/supplementary/skyscraper/Skyscraper -p amiga --nosubdirs --cache report:missing=all

I get results for all files and not just the ones in the target directory. It would be really useful if I could limit the report down to only files that I want to target for gathering.

muldjord commented 5 years ago

Thank you for reporting this. 3.2.7 now released where this is fixed. It will now adhere to the --nosubdirs option as expected. And of course you can also use the -i <FOLDER> option in addition to this, if you just want to check other folders than the base platform rom folder. But I assume you are aware of that option already, just wanted to mention it.

Have fun!

vanativo commented 5 years ago

@muldjord , I updated to 3.2.7 to give it a try but I'm not getting the results I'm expecting, unless I'm misreading them or not using the option correctly of course.

So if I run:

/opt/retropie/supplementary/skyscraper/Skyscraper -p c64 --cache report:missing=description,publisher,developer,releasedate,marquee,video --nosubdirs

I get:

Reading and parsing resource cache, please wait...
Successfully parsed 4499 resources!

Creating report(s) for resource type(s):
  description
  publisher
  developer
  releasedate
  marquee
  video

Input folder returned no entries...

0 compatible files found for the 'c64' platform!
Creating file id list for all files, please wait...

Report filename: '/opt/retropie/configs/all/skyscraper/reports/report-c64-missing_description-20190901.txt'
Assembling report, please wait... Done!
0 file(s) is/are missing the 'description' resource.

Report filename: '/opt/retropie/configs/all/skyscraper/reports/report-c64-missing_publisher-20190901.txt'
Assembling report, please wait... Done!
0 file(s) is/are missing the 'publisher' resource.

Report filename: '/opt/retropie/configs/all/skyscraper/reports/report-c64-missing_developer-20190901.txt'
Assembling report, please wait... Done!
0 file(s) is/are missing the 'developer' resource.

Report filename: '/opt/retropie/configs/all/skyscraper/reports/report-c64-missing_releasedate-20190901.txt'
Assembling report, please wait... Done!
0 file(s) is/are missing the 'releasedate' resource.

Report filename: '/opt/retropie/configs/all/skyscraper/reports/report-c64-missing_marquee-20190901.txt'
Assembling report, please wait... Done!
0 file(s) is/are missing the 'marquee' resource.

Report filename: '/opt/retropie/configs/all/skyscraper/reports/report-c64-missing_video-20190901.txt'
Assembling report, please wait... Done!
0 file(s) is/are missing the 'video' resource.

All done!

But I happen to know this is not the case, there are plenty of missing resources in my C64 cache. For example, look at this entry from my current game list:

 <game>
    <path>/home/pi/RetroPie/roms/c64/Alternative World Games.d64</path>
    <name>Alternative World Games</name>
    <cover />
    <image>/home/pi/RetroPie/roms/c64/media/screenshots/Alternative World Games.png</image>
    <marquee>/home/pi/RetroPie/roms/c64/media/marquees/Alternative World Games.png</marquee>
    <video />
    <rating />
    <desc />
    <releasedate>19870101T000000</releasedate>
    <developer />
    <publisher>Gremlin Graphics</publisher>
    <genre />
    <players />
  </game>
muldjord commented 5 years ago

It says that it found 0 compatible files. I need more info. You are using the --nosubdirs option. This means it will only look for files in the c64 rom folder (probably /home/pi/RetroPie/roms/c64) and NOT any subfolders. So if all your roms aren't directly in that folder, it won't find them. If you want it to look into a specific subfolder you need to use -i <FOLDER>. I've tested it and it works as expected on my end.

vanativo commented 5 years ago

Hmmm. Well, the interesting thing here is that I haven't changed my files from when I did the resource gathering which worked. So something seems inconsistent between the resource gathering and the cache reporting.

My setup looks like this: At the top level which is /home/pi/RetroPie/roms/c64 (the default directory for c64 roms) I have symbolic links to the actual rom files that are in subdirectories. For example:

ls -l

drwxr-xr-x   2 pi pi   4096 Aug 23 20:45 Dragon's Lair
lrwxrwxrwx   1 pi pi     84 Aug 23 20:46 Dragon's Lair.d64 -> /home/pi/RetroPie/roms/c64/Dragon's Lair/Dragon's Lair (1986)(Software Projects).d64

The following is successful:

pi@retropie:~ $ /opt/retropie/supplementary/skyscraper/Skyscraper -p c64 -s screenscraper -u XXXX:XXXX --videos --nosubdirs "Dragon's Lair.d64"
------------------------------------------
Running Skyscraper v3.2.7 by Lars Muldjord
------------------------------------------
Platform:           'c64'
Scraping module:    'screenscraper'
Input folder:       '/home/pi/RetroPie/roms/c64'
Game list folder:   '/home/pi/RetroPie/roms/c64'
Covers folder:      '/home/pi/RetroPie/roms/c64/media/covers'
Screenshots folder: '/home/pi/RetroPie/roms/c64/media/screenshots'
Wheels folder:      '/home/pi/RetroPie/roms/c64/media/wheels'
Marquees folder:    '/home/pi/RetroPie/roms/c64/media/marquees'
Videos folder:      '/home/pi/RetroPie/roms/c64/media/videos'
Cache folder:       'cache/c64'

DID YOU KNOW: You can scrape a single game by adding it to the command line either with partial or full path. You can also scrape a span of games with the '--startat' and '--endat' command line options.

Fetching limits for user 'Vinny', just a sec...
Setting threads to 5 as allowed for the supplied user credentials.

Reading and parsing resource cache, please wait...
Successfully parsed 4500 resources!

Looking for optional 'priorities.xml' file in cache folder... Not found, skipping...

Trying to parse and load existing game list metadata... Success!

Starting scraping run on 1 files using 5 threads.
Sit back, relax and let me do the work! :)

#1/1 (T1) Pass 1 ---- Game 'Dragon's Lair' found! :) ----
Scraper:        screenscraper
From cache:     NO
Search match:   100 %
Compare title:  'Dragon's Lair'
Result title:   'Dragon's Lair' ()
Platform:       'Commodore 64' ()
Release Date:   '1986-01-01' ()
Developer:      'Cinematronics' ()
Publisher:      'Software Projects' ()
Players:        '1' ()
Ages:           '' ()
Tags:           'Action' ()
Rating (0-1):   '0.65' ()
Cover:          YES ()
Screenshot:     YES ()
Wheel:          YES ()
Marquee:        YES ()
Video:          YES ()

Description: ()
'The first port of the popular arcade game Dragon's Lair to a home computer, featuring the infamous Dirk the Daring.

The game, as one would expect from the long-running Dragon's Lair series, has mixed gameplay, including platform, puzzle and arcade (in the old-fashioned choose-your-path sort of way).

This first port includes two adventures: the first is to raid the lair of Singe the Dragon (hence the classic name) and rescuing Princess Daphne, and the second, in a sort of instant sequel of the first chapter, features Dirk trying to escape from the castle, now overrun by the Lizard King and its minions.'

'screenscraper' requests remaining: 19969

1/0/1
Elapsed time   : 00:00:17
Est. time left : 00:00:00

---- Resource gathering run completed! YAY! ----
Writing 4500 (0 new) resources to cache, please wait... Success!

---- And here are some neat stats :) ----
Total completion time: 00:00:17

Average search match: 100%
Average entry completeness: 92%

Total number of games: 1
Successfully processed games: 1
Skipped games: 0 (Filenames saved to '~/.skyscraper/skipped-screenscraper.txt')

So that implies it found the file I specified as being a valid target in the default C64 directory since I didn't provide a path. However the cache command is not identifying the same file as being valid. Is it possible that the cache command is not recognising symbolic links while the normal gathering command is?

muldjord commented 5 years ago

That's the reason. I will make it include symlinks as well. Stay tuned.

muldjord commented 5 years ago

3.2.8 out now which supports symlinks for the cache features.

vanativo commented 5 years ago

Wow fast work!!! I will try it now and let you know the results.

vanativo commented 5 years ago

That's got it!!! Thank you again! Great work.

muldjord commented 5 years ago

Awesome! You're welcome and thank you :)