Add support for editing Stardew Valley maps

[bjorn]

Stardew Valley maps are stored inside .xnb files (produced by Microsoft XNA Game Studio) in .tbin format (a binary tile map format used by tIDE). It should be possible to write a Tiled plugin that can directly load and save such maps.

• SimpleXNBDemapper shows how .tbin and .png files can be unpacked from (and packed into) the .xnb files (see Program.cs)

• TbinFormat.cs shows how a .tbin file can be parsed

On the Tiled forum somebody reported to have successfully extracted the necessary .tbin and .png files and then used tIDE to open the map and save it as .tmx so that it could be opened in Tiled. It would be cool to simplify this process by supporting .tbin and maybe even .xnb files containing .tbin files directly.

[spacechase0]

Some other things:

• Most (all?) of the game's XNBs are compressed (using LZX), which SimpleXNBDemapper can't handle. I did recently find this. I tested it on Data/eventConditions.xnb and it appears to have worked successfully. Their C# decompression is mostly just a port of libmspack. (It says dual-licensed LGPL/MS-PL. Not sure how that works since libmspack is only LGPL.)
• The project above (and the original libmspack) doesn't support compression. This isn't a huge issue though, since (as SimpleXNBDemapper mentions) XNBs are valid even when not compressed.
• We're currently using a modified version of tIDE to export a .tbin to .tmx. (Mentioned and briefly explained here). I don't know the full details of the modifications but I believe @Pathoschild was given the source code.
• Stardew Valley maps heavily rely on individual tile properties, which it seems like Tiled doesn't support? (Or I'm just missing them.) Right now we have an object called "TileData" precisely on each tile that has properties.
• .tbin animations only support a single frame interval per set. I don't know how our tIDE modification handles different animation lengths.
• The .tbin tilesheet textures don't even list an extension, just "paths" or "spring_outdoorsTileSheet", due to the way the XNA content pipeline works (I think).
• The current modding API does allow us to make maps using both our own tilesheets (which can be .png if you include the extension in the image source) or the base game tilesheets. So it'd be reeeeeeeeeeeeeally handy if we could have multiple search directories for tilesheets.

If recompression is extremely important to you, I have started writing a LZX de/compressor for some of my own projects, and I'm just going to throw it under MIT (if I finish it).

That's all I've got for now.

[spacechase0]

So I got my XNB loader working for most of the Stardew files (and certainly the ones Tiled would need). It's here. I re-ported the library I mentioned earlier here.

If you aren't happy with that license for that compressor, someone apparently wrote their own here. (It doesn't have a license right this moment, but the author tells me he'll put one up in the next day or two. He mentioned "MIT or whatever" earlier today.) (GPL)

I'm probably going to be writing a tbin loader soon to feature-creep my own project. I'll throw that under MIT as well.

[bjorn]

@spacechase0 That's so awesome! Let me know if I can help in any way. Right now, I've got a lot of other things with higher priority, like fixing things for Tiled 1.0.1, mentoring the GSoC students and updating the manual, but I'd really like to work on this as well.

[spacechase0]

I got stuck for a while since I didn't know func( a(), b() ) didn't guarantee the execution order of a and b, but my Tbin loader is done now.

[spacechase0]

It looks like I'd just need to create something similar to the Json plugin, right?

I noticed that nothing uses the STL. Should I port the things I already wrote to Qt classes, or is the STL fine? (I assume either way I'll be putting these as a sub-directory in the plugin?)

I see that there is a way to add support for another image format with a QImageIOPlugin. How reasonable would it be to implement it this way? Or would you prefer a more "manual" approach? (If the latter, some guidance would be appreciated. I keep getting lost trying to find things in the codebase.)

[bjorn]

It looks like I'd just need to create something similar to the Json plugin, right?

At this point, you could import this .tbin loader and turn it into a Tiled plugin. For that it would need to be stripped from its SFML dependency and the logger statements need to be changed.

Note however that there are some differences between tIDE and Tiled that may be problematic:

• tIDE appears to support different tile size per layer, Tiled does not. Hopefully SV doesn't use this feature.
• tIDE appears to store properties for each tile location, which Tiled doesn't directly support (for performance reasons). When loading, objects would need to be created to store these properties and when saving would need to be serialized back to the tile layer.
• tIDE appears to store full tile animations instead of tile indexes in animated locations, and different frames of an animation can come from different tilesets. In contrast, Tiled defines an animation on the tile and refers to it by tile index as usual, and it can only use frames from the same tileset. This means the loader would need to add new tiles for each unique animation encountered, and animations with frames from different tilesets can't be supported (though care could be taken to keep them intact upon saving by using additional meta-data).

I noticed that nothing uses the STL. Should I port the things I already wrote to Qt classes, or is the STL fine? (I assume either way I'll be putting these as a sub-directory in the plugin?)

Using STL is fine. Actually std::map is used by RangeSet. :-)

Using a subdirectory would be good indeed.

I see that there is a way to add support for another image format with a QImageIOPlugin. How reasonable would it be to implement it this way? Or would you prefer a more "manual" approach? (If the latter, some guidance would be appreciated. I keep getting lost trying to find things in the codebase.)

I'd say we should implement just the .tbin support at first, since it will already help people, and we can then indeed look at adding this.

[spacechase0]

At this point, you could import this .tbin loader and turn it into a Tiled plugin. For that it would need to be stripped from its SFML dependency and the logger statements need to be changed.

That shouldn't be hard. The SFML stuff is just sf::(Ui|I)nt(8|16|32) which is always (unsigned)? (char|short|int) anyways.

The logger stuff was mainly for my debugging. There used to be a lot more. :P Do you want me to transition those to Tiled or just remove them?

tIDE appears to support different tile size per layer, Tiled does not. Hopefully SV doesn't use this feature.

It doesn't.

tIDE appears to store full tile animations instead of tile indexes in animated locations, ... (though care could be taken to keep them intact upon saving by using additional meta-data).

The game itself always keeps animations on the same tileset, so unless you want "full" tbin support we can ignore that.

[bjorn]

The logger stuff was mainly for my debugging. There used to be a lot more. :P Do you want me to transition those to Tiled or just remove them?

For as much as the logged output is actually needed for user feedback about errors, it could be changed to error codes that lead to translatable strings in the Tiled plugin. The parts that are only for debugging should probably just be removed.

[spacechase0]

This is probably pretty simple, but I'm getting an error that says "Referenced directory 'G:/Forks/tiled/src/plugins/tbin' does not contain a qbs file." after editing plugins/plugins.qbs. Going to the directory does in fact have a qbs file. Any ideas?

[bjorn]

The name of the file may need to match the folder name. It is called tbin.qbs?

[spacechase0]

It does.

I don't know if this is related or a separate issue, but the folder doesn't show in Qt Creator either.

https://github.com/spacechase0/tiled/commit/fab57f8a386c26d03eba766e30429db8a467a032

[bjorn]

@spacechase0 I got this when I tried to run qbs:

[bjorn@thor tiled-pr]$ qbs build
Restoring build graph from disk
Resolving project for configuration default
ERROR: /home/bjorn/projects/tiled-pr/src/plugins/tbin/tbin.qbs:8:9 Unexpected token `string literal'

It is due to a missing comma after the first file name in the list.

Btw, please don't hesitate to open a pull request early. It makes it easier for me to help with such problems as well.

[bjorn]

Thanks to @spacechase0, there should now no longer be a need to go through tIDE to edit Stardew Valley maps. I tested it on some extracted .tbin files and it seemed to work very well! The new plugin is now included in the development snapshot.

@spacechase0 Do you think it makes sense to continue with actually supporting opening maps directly from .xnb files? I guess with your XNB loader it would be no problem to do it for the maps, but it may be hard to make it work for the tileset images (though the plugin could load them in some custom way I think - probably better than doing a Qt image IO plugin). But if in general people anyway need to unpack and repack their data then loading directly from .xnb files isn't that useful.

[bjorn]

Note to self: don't forget to add the tbin plugin to the Windows MSI installer.

[spacechase0]

It could be convenient in some cases, but it definitely is not necessary. People who only edit the map would fine it useful, but those who edit the tilesheets do still need to repack the images. (And those of us who make new maps for SMAPI mods don't need to pack either.)

[bjorn]

@spacechase0 Then I'll close this issue. Thanks a lot for your work on this!