search

bog-dan-ro / spectacol

ZX Spectrum Emulator Based on FUSE with a custom UI made with QML
42 stars 3 forks source link

Games metainfo #34

Open bog-dan-ro opened 8 years ago

bog-dan-ro commented 8 years ago

Currently Spectacol only displays the game file name and the game screen (if available) but it doesn't have proper support for metainfo like:

I'd like Spectacol to access that information from a remote service but also to be able to contribute to that database, to be able to add/edit and even remove metainfo.

ZxAsSeMbLeR commented 8 years ago

Good idea!. The main problem is how to make use of the already available info out there, but even if each user have to build his own database (with the possibility of sharing it with other users) it would be really nice.

bog-dan-ro commented 8 years ago

I'll try to convince WOS to share it: http://www.worldofspectrum.org/forums/discussion/52464/is-infoseek-database-available#latest

ZxAsSeMbLeR commented 8 years ago

That would be really nice!

Moreover, it is always interesting to have the possibility for accessing that information locally. When dealing with other emu a while ago (Speccy Deluxe), I adapted the covers and screenshots of an already existing database for Gamebase in Windows, called Speccymania to make use of them in Speccy Deluxe using the format and syntax required to show the box cover/screenshot. I made that using scripts and batch processes, and nearly 70% was ready and perfectly visible with that emu from scratch. The other 30% (debugging errors not correctly mapped by the automatic process) is a slow process I am still in ;-) (dedicating 5 or 10 minutes a day, maybe in 3 months I will have more than 12000 games with covers and screenshots vinculated). The format used in that emu is:

Let's suppose the file is uncompressed:

Filename of the game: . Filename of the screenshot: ..png Filename of the boxcover: ..box

If the file is compressed all three files go into a single .gz file

Bearing in mind that there are LOTS of files involved, I have them splitted into a lot of folders and subfolders. First 123, A, B, C..., then in a second level in subfolders of 100-150 games each subfolder renamed with the name of the first game of the group and the last one of that group. That way the games can be located easily and browsing them with all the info is fast.

Maybe something similar could be used to see more screenshots and/or box covers, for example. Browsing in realtime could be slow, so maybe they could appear when pressing somehow the selected game and selecting the right option.

One curiosity....how do you manage to put the screenshots so fast? are they available somehow in some of the formats (in z80 they are always there, but that is not the case in other formats)?

Thanks!

ZxAsSeMbLeR commented 8 years ago

The syntax has been "cut" in the previous message:

Filename of the game: "name of game"."extension" Filename of the screenshot: "name of game"."extension"."png" Filename of the boxcover: "name of game"."extension".box

bog-dan-ro commented 8 years ago

Check https://github.com/bog-dan-ro/spectacol/blob/master/fuse/ui/qml/spectrumscreen.cpp to see how I'm getting the screenshots.

I'm working on a meta format for any ZX files. It is similar with tzx but it's based on protobuf which allows to be extended and keep back and forward compatibility, which, sadly, tzx doesn't have.

I'll publish it soon (hope to have time this weekend) to hear the community suggestions and feedback ;-)

ZxAsSeMbLeR commented 8 years ago

Ok, now I understand it better. So, you search the file data blocks and if there's a screen block (detected by the size of the block), you pass it to the code to draw that screen. Then a screen block must be present in the file to be able to draw it (otherwise the tape picture is painted).

We will look forward to hearing from that meta format. Just advice us when you publish it ;-)

bog-dan-ro commented 8 years ago

Published the first draft here https://gist.github.com/bog-dan-ro/dd463a7d5f7d94aaa933d412a68b00de And opened a discussion about new features to include in next major version here: http://www.worldofspectrum.org/forums/discussion/52504/spectacol-2-0-new-features

ZxAsSeMbLeR commented 8 years ago

I will have a look at the draft. Thanks for sharing!

You are right about the zip format comment...anyway the reason it could be desirable to have support for it is because of large zipped collections out there (including personal collections). Yes, one could really decompress and compress again in other format...;-)

ghost commented 8 years ago

There is a offline database file here that contains Title, Publisher, Year, CRC etc.

http://www.worldofspectrum.org/rainbow-wos.zip

It was linked in the ZX-Explorer page at http://www.zx-modules.de/

I'm not sure the last time they updated it but would this only recognize WOS game versions and not TOSEC etc. ? WOS have a lot of games missing due to "Distribution Denied". (And probably doesn't link to Screenshots etc. which would be very complicated IMO).

bog-dan-ro commented 8 years ago

The easiest way to get the metadata is to parse the TZX files which (at least on wos) have most of the needed information. They are missing the pokes, instructions & magazine scans, but if I'll manage to create a nice editor in Spectacol, people will be able to contribute by updating them easily.

bog-dan-ro commented 8 years ago

I have some updates on this feature: First and foremost I'm going to drop the protocol buffers and use flatbuffers instead. Flatbuffers offers the same flexibility, but the speed is way much better.

Another big change is that, the new format is not just a metainfo, but a container, this means that it will actually contain also your ZX files.

I also split the file in 3 pieces:

  1. https://gist.github.com/bog-dan-ro/910cc13f6a5ff2f2649788328717db16 is the common part for both server & emulator
  2. https://gist.github.com/bog-dan-ro/dc9d16fe9f72a08e24b57590c883b964 is what it will be stored in database on the server side
  3. https://gist.github.com/bog-dan-ro/51fdcd772594ebb317468bcda71dc2ff is what the emulators will use. This format has two more entries, the snapshots & movies. These are used to store users snapshots & movies while is playing games., therefore they are not needed in the server database. Even the format seems different, the emulator will be able to open and use any files sent by the server even they are missing the two entries! Same for the server, if an emulator sends a message with these two entries, the server will be able to read what it needs and discard the rest of it.

It will be great if anyone can check these file to see if anything is missing

The plan is to create:

This feature has the highest priority for 3.0 release (besides the bug which must be squashed).

bog-dan-ro commented 8 years ago

@windale thanks a lot for rainbow-wos.zip suggestion, it helped me to create a nice ZXDB from it ;-). Now I just need to figure out how to merge it with the archive.org one , some titles are not the same ...

ZxAsSeMbLeR commented 8 years ago

Everything is more or less covered. At first sight, maybe I would add: "SNA" to "ZXFileType" and I would differenciate also "Map" in "ScanType".

Merging database entries with filenames is not so easy. I did that with all the games inside a database+games called Speccymania, just to match all covers and screenshots with the real names inside the zip files and not the zip filenames themselves (as the database covered). Some could be made programatically but other was made by hand.

bog-dan-ro commented 8 years ago

Ah, I knew I forgot something :) . I think I'll create a separate section on table File {..} for maps and I also move rzx next to it. Regarding SNA ... it is handled in the snapshots section on the emulator side. IMHO Z80 is enough for the database (unless SNA is a superior format).

Thanks for checking it, I'll update them in ~2 weeks.

bog-dan-ro commented 8 years ago

I updated the format. I order to log the history and I created a new repository here https://github.com/bog-dan-ro/zx_format