Suggestion: an utility for un/repacking sprite file

[ghost]

Hello. In the past I was considering a theoretical use of agsutils by a game developer to setup a patching system for their own game, and the remaining problem is a sprite file (acsprset.spr). In big games it may reach gigabytes in size. Currently agsutils allow to extract it from the game, but (afaik) not extract/repack its contents.

Hence, I think an utility to unpack/repack acsprset.spr could be a useful addition.

The related reading/writing code may be found here: https://github.com/adventuregamestudio/ags/blob/ags3/Common/ac/spritecache.cpp#L578 Note, the file comes in 2 variants: with compressed and uncompressed bitmaps. Only supported compression is RLE (there were user requests to add other options, which may occur in future).

[rofl0r]

this is a good idea. i actually already implemented a prototype of it locally, which works with 32 bit bitmaps already, but i have one major issue: newer spritecache formats don't save the palette, but their 8bit images are raw pixel data, without any palette associated. i suppose after glancing over the engine code, that the palettes are loaded via script from another location (probably main "pack" file). do you have an idea how to get the right palette for such images?

also, what is the native image format ? is any image imported immediately added into the spritecache format, or does it exist in another image format (bmp?) in the game source tree, until an exe is produced ?

[ghost]

newer spritecache formats don't save the palette, but their 8bit images are raw pixel data, without any palette associated. i suppose after glancing over the engine code, that the palettes are loaded via script from another location (probably main "pack" file). do you have an idea how to get the right palette for such images?

This is indeed complicated, because in latest versions final palette is a merge between "default" and "room" palettes. Game struct contains an array called paluses, this array defines which slots are game-wide and which are room-specific. When room is loaded the room slots are overwritten. Startup code: https://github.com/adventuregamestudio/ags/blob/release-3.5.0/Engine/main/engine.cpp#L871 Room code: https://github.com/adventuregamestudio/ags/blob/ags3/Engine/ac/room.cpp#L496

It's also possible to modify palette in script, that's used for various dynamic effects.

The game's default palette is stored as GameSetupStructBase::defpal and room's palette as RoomStruct::Palette.

When importing a sprite (in the editor) user also has an option to remap palette using currently loaded room (I think this is used for room's objects), but afaik this converts imported image at once and choice itself is not saved into compiled data.

also, what is the native image format ? is any image imported immediately added into the spritecache format, or does it exist in another image format (bmp?) in the game source tree, until an exe is produced ?

Currently any image is converted into raw bitmap (Allegro's bitmap format) on import, then saved to the sprite file (acsprset.spr) when game project is saved. Editor also stores meta-data about sprite's source (in Game.agf, which is a big xml file) and allows user to re-import sprites from the source using same settings. Since AGS 3.5.0 Editor also allows to change the sprite's import settings at will (previously one would have to fully replace sprite instead).

It's planned to change this workflow in future though and keep original images at design time, only creating/ammending acsprset.spr as a part of final game compilation.

[rofl0r]

@ivan-mogilko i added a prototype for extraction ^, would you mind giving it a try and some feedback? thanks!

[ghost]

Sorry, I haven't had time to test it out, but wanted to mention my concern about TGA output. I don't know how much this is used overall, but it seems to be not fairly used around AGS community (some people I asked did not even know what it is). In fact, AGS Editor does not provide TGA option when importing sprites, which means users would have to do conversion if they'd want to reimport extracted sprites into AGS project.

Could you consider PNG instead, for instance?

[rofl0r]

I don't know how much this is used overall

it seems to be supported by all picture viewer/editing sw i'm aware of

Could you consider PNG instead, for instance?

the reason i opted for tga is that it's dead simple and can be implemented in a reasonable amount of time without pulling in 3rd party libs, and that it's supported by all popular image editing sw. also the RLE compression i implemented yesterday seems to be rather effective: disk size for extracted sprites from Quest for Infamy shrank from 800ish to 320 MB (original compressed acsprset.spr: 385MB).

AGS Editor does not provide TGA option when importing sprites, which means users would have to do conversion if they'd want to reimport extracted sprites into AGS project

my idea of how this is supposed to be used is mainly for people like @raulpuro who want to do translations or modifications, so they could extract the sprites, edit it e.g. in gimp or photoshop, then reimport it into the spritepack file. or people that want to do a remake of some classic, but are capable of using a command line tool like imagemagick convert to do a batch-convert job. after all, these utils are command line tools themselves.

tbh, it'd be way easier to add tga reading code to AGS than to implement PNG format by hand here in agsutils.

[ghost]

Alright, I made a first test of agsprite, and found a bug: it seems that the last sprite is not extracted. For example, trying my "Last'n'Furious" game (https://www.adventuregamestudio.co.uk/site/games/game/2216/), the last sprite is 105, a grey racer portrait, but it is not extracted.

Another thing, agstract utility creates an output directory if that does not exist, but agsprite does not; would it be possible to support that in agsprite too?

PS. Regarding images themselves, I tried two editors - GIMP, and something called Pinta (I've read about it in this recommendation article). GIMP opened exported TGAs successfully; Pinta complained about unknown format, but maybe it's editor's issue.

PPS. Are you planning to support packing the sprites back too?

[rofl0r]

found a bug: it seems that the last sprite is not extracted.

thanks, fixed

agstract utility creates an output directory if that does not exist, but agsprite does not; would it be possible to support that in agsprite too?

yes, implemented now.

I tried two editors - GIMP, and something called Pinta (I've read about it in this recommendation article). GIMP opened exported TGAs successfully; Pinta complained about unknown format, but maybe it's editor's issue.

pinta is probably not the best choice, it depends on mono which is a huge framework, iirc about 300 MB disk use, and i dont think it's widely used. the sad truth is that there aren't any good photo editing programs for linux: gimp is powerful, but with an utterly counterintuitive UI. most others that are usable have totally unreasonable dependencies (such as mono or in case of krita, the entire KDE framework + QT). one that's quite good is mtpaint, though it's UI is optimized for pixelgraphics, not photos.

there are some good viewers though, for example sxiv or feh which will support tga with palette as soon as my patch has landed in imlib2. gpicview also supports tga but is really slow when scrolling through the picture collection.

on windows, afaik all common editors (corel, ps, paint.net...) and viewers (irfanview...) support the format.

Are you planning to support packing the sprites back too?

yes

[rofl0r]

Are you planning to support packing the sprites back too?

this is now implemented. as mentioned in the commit message ^ there's an issue where the menu selection bar in last'n'furious isn't visible. it would be very helpful if you could try this out (you need to remove sprindex.dat as described in the commit message), as with your engine know-how you probably can immediately figure out what's wrong. i'd be glad if you could test this relatively quickly so i can stay "in the flow", thanks!

[ghost]

there's an issue where the menu selection bar in last'n'furious isn't visible

I tried it, and found that the sprite has lost its alpha channel after repacking, every pixel has alpha = 0, while the original sprite had transparent pixels with alpha 0 and non-transparent with alpha 0xFF.

Unfortunately, AGS stores "has alpha channel" flags in a separate data inside the game struct, not sprite file.

To be fair, that particular sprite in "L'n'F" was incorrectly imported and was not supposed to use alpha channel at all (but rather have a regular "magic pink" transparent color), but idk if that matters. I will try a game that actually intentionally makes use of alpha channel a bit later.

On a side note, if I understood correctly agsprite produces a compressed pack; would it be possible to make this optional too?

[rofl0r]

I tried it, and found that the sprite has lost its alpha channel after repacking, every pixel has alpha = 0, while the original sprite had transparent pixels with alpha 0 and non-transparent with alpha 0xFF.

thanks, all alpha-related issues should be fixed now.

Unfortunately, AGS stores "has alpha channel" flags in a separate data inside the game struct, not sprite file.

that's indeed unfortunate. i added a tool "agsalphahack" to be able to fix this manually, if desired.

On a side note, if I understood correctly agsprite produces a compressed pack; would it be possible to make this optional too?

yes, i added such an option now.

[rofl0r]

imo agsprite is now feature-complete, so this can be closed. further testing would be welcome @ivan-mogilko