Linux and the Content Pipeline

[3vi1]

It seems like the only real obstacle to using LDEngine while using Linux as our development platform is the missing content pipeline feature used for the Tiled map support. I was happy to see (via a Tweet @ me) that Gareth seemed receptive to the idea of possibly adding support for TMX loading separate from the pipeline, as I do all of my development goofing around from Linux.

I don't know if you already have specific plans or code written towards this goal, but if not I'd be happy to take a look/stab at it. I haven't looked at your map classes yet, but it seems like it might be relatively easy.

Earlier today, I threw together a fork (https://github.com/3vi1/LDEngine-FNA.git) with some files to start building everything from Linux. It adds some MonoDevelop Linux sln/csproj files, and includes Ethan Lee (flitijibibo)'s MonoGame-SDL2 (soon to be FNA) as a ThirdParty submodule. I prefer to use MonoGame-SDL2 because it generally supports the Linux and Mac desktops better than upstream MonoGame.

The fork will build and run the LDEngine demo on Linux right now (using a pre-built map.xnb), with a few caveats/notes:

• There is a separate Linux content project, which uses the MonoGame Content pipeline to build the PNGs and spritefont. The font is in this directory with the correct name, as MonoGame's pipeline does not look at installed fonts.
• You will need to right-click the content project and build it before attempting to build LDEngine for the first time. This is due to a shortcoming of MonoGame content project files: "the ability to add a Content project to a MonoGame project is NOT yet supported". I tried to hack around it by playing with references, but it just doesn't work right now.(https://github.com/mono/MonoGame/wiki/Xamarin-Studio-Content-Pipeline-Addin)
• You may see a compile warning: "Problem processing Content Extensions Object reference not set to an instance of an object (LDEngineContent.Linux)". This appears to be due to another bug in the MonoGame content pipeline, but it still produces the XNB's... so I'm not worrying about it too much yet.
• You may even see an nvtt SystemEntryPointNotFoundException (or something like that) when building the content project clean. This is yet another content pipeline bug (that I though was fixed, yet I still see it even when I compile upstream MonoGame and replace the content project dlls with the newest).

[GarethIW]

This is really awesome, and it's great that you've decided to take my little engine and work with it!

I wasn't planning on doing any more work on LDE for now as I have a big game project to get back to, but I'd be happy to help out if I can and look at any PRs you throw my way.

I'm not a Linux user in general, I build my Linux ports in VS on Windows by way of linking the compiled .xnbs as content. I've not looked at the Monogame CP as it's always been in an incomplete state. And I'm not really familiar with Ethan's fork, apart from marvelling at how awesome it is and hoping that the MG devs get around to merging the SDL branch in someday (looking less and less likely).... so I'm not sure how much help I can be!

Most of the Tiled lib is Nick Gravelyn's original work. I added in the fetching of colour data for selected tilesets to allow for the per-pixel detection, and I added in (hacky) support for PolyLines, but that's about it. I've never really liked having to have the TileContentPipeline project in my solution and always thought I'd rather load the raw tmx directly (with a static Map.Load(filename) and loading the tilesheet pngs from the same directory maybe). Should just be a matter of moving the pipeline processing code to a Map.Load function.

More stuff that needs to be added:

• Proper support for Polygon, PolyLine and Ellipse objects
• Support TileSet properties (Tiled didn't used to have a properties collection on a tileset, which is why the Collision stuff requires a property to be set on the first tile of a sheet)

[3vi1]

Coolio. I'll try to have a look at your map code this weekend then, and see what I can do with it.

It will be nice to get this working smoothly with a Linux development platform, because then I won't be tempted to recreate the wheel and make a second-class version of all these things that you've obviously already put a lot more thought and time into than I.

[3vi1]

I have this mostly working. I just need to squash one bug and clean up the code a bit.

First I tried re-creating the wheel, as I am wont to do. it had appeared that it would be quick to write a thin wrapper to load the XML, and have a new TiledLib constructor with the ability to read that. After several hours the scope kept growing as it became evident that the map schema doc (https://github.com/bjorn/tiled/blob/master/docs/map.xsd) was out of date with the current file format and I was tweaking the code that I auto-generated from the schema way more than expected. Just as I found myself adding zlib support, it struck me that I was about 80% of the way to recreating TiledSharp.

After coming to my senses, I added a couple of TiledLib constructors that will work with TiledSharp.

I uncovered a couple of bugs with TiledSharp along the way though. The nastiest of which is that it assumes there's only one tile in your tileset if each tile is not explicitly declared in the .tmx file (which Tiled does not do - it only adds the ones that have properties set). I'm going to submit a fix to the upstream TiledSharp project that infers the existence of the additional tiles when the tilewidth is smaller than the image width to fix that.

I've got family stuff planned for the rest of the day, but I should have this all sorted soon. I'll probably throw some pull requests your way next week.

[3vi1]

For the new TMX-loading TiledLib constructor, I updated the collision flagging such that you can either use a CollisionSet property on the tileset itself, or the same property name on individual tiles within the tileset. This allows you to have one tileset with both collideable and non-collidable tiles, and the loader will only build bitmasks for the collidable portions. For instance, I've added the property just to tile 8 in the demo, and it now only caches those 256 bits instead of the 4.5Kbits of the whole texture.

The CollisionSet property is inherited from tileset to tile, but the tile's value will override that of the parent tileset. This also allows you to specify the entire tileset is collidable, then override that value to make specific tiles immaterial.

Tileset files should be fully supported/working now too.

I still need to go back and fix that in the original XNB-loading TiledLib constructor. I didn't add the improvements there, and I've probably broken collision for XNB's now that detection uses the actual TileWidth when computing the pixel offset instead of that of the entire texture. I'll make this repair my next step, as I suspect you might want to keep the ability to load XNBs for backwards compatibility.

[GarethIW]

Wow, thank you so much for your work on this. Really looking forward to not needed the pipeline project any more :) But equally, it's good that you're keeping the original XNB support for backwards-compat.

Love the idea of per-tile collision flagging - I always forget just how much memory color data takes up!

Looking forward to your PRs. I've probably got a few bits to add/change myself since actually using the engine in LD29. I keep getting suckered into more gamejams though, so finding the time will be fun :)

[3vi1]

Coolio. I got XNB loading fixed on my commute home yesterday. It doesn't have the per-tile collision improvements, but it should work the same as it always did for all your previous projects.

I've re-forked your project so that I can commit/submit these additions in more clean and easily digestible chunks. Hopefully I can start the submissions tonight.

One question: Do you have a specific preferences when it comes to file formatting? I noticed that the first half of TiledLib/Map.cs is using spaces and the second half is using tabs (which, in my IDE are not set to four spaces), so I'd like to clean up the formatting to be consistent as part of my patch. The formatting I'm generally using these days matches https://github.com/flibitijibibo/MonoGame/issues/147 (I got used to it while cleaning up the MonoGame code), but I will use whatever you plan to use for this project to keep it all consistent.

If you can let me know about your preferences on that, I will plan to break the patches up in separate commits for 1) formatting, 2) Tmx Loading + TiledSharp submodule additon, 3) Linux MonoDevelop solution/project files + MonoGame-SDL2 submodule addition. The submodules will be all kept separate in a top-level ThirdParty/ directory (like MonoGame-SDL2 does with its submodules) in order to keep them all distinct from the main project.

[GarethIW]

I use and prefer VS/C# standard of 4 spaces, but as there's plenty of copypasta code from various old projects I imagine there's some inconsistencies. It's not something that usually bothers me.

Also, submodules confuse the hell out of me, so I might need some help merging if it involves getting deep into Git CLI :) I'll be more than happy adding you as a contributor if needs be.

[3vi1]

Spaces/VS defaults it is, then!

Don't sweat the submodule stuff. I'm no git guru, so we'll both get some experience here.

That said, I expect this to be really simple: My submodule additions are bundled in a pull request (as a change to .gitmodules). The only thing that anyone building the engine will need to do is run two additional commands after cloning the repo:

git clone https://github.com/GarethIW/LDEngine.git
cd LDEngine
git submodule init
git submodule update

(People that already have a local repo only need to run the last two commands, after pulling the commit with my .gitmodules update, of course.)

...and that tells git to set up the submodules in your local copy and pull in the latest versions of those projects to the ThirdParty/ directory. From then on, devs can stick with those versions, or occasionally execute git submodule update to get the latest+greatest commits to the thirdparty projects.

I'm a bleeding-edge junkie, so I like using submodules to constantly grab the latest improvements to outside components - and it makes it really easy to debug an issue all the way into upstream/thirdparty code.

[3vi1]

Just realized, while building LD29 on the van this morning, why you had the "#if WINDOWS || XBOX360" in your program.cs. We'll want to add that back and include LINUX as well (or just add "#if !MACOS" instead, if I remember Icculus's rules right from his Valve talk) so that building on Mac works as you originally planned. :tongue:

Hopefully this weekend I'll get a chance to actually read the LD29 code and start working on something of my own using LDEngine.