Feat/29/dt1 decoder

[grantramsay]

29 This sounds cool mixing Diablo 1 & 2 textures!

I had a quick go at porting the Riiablo DT1 decoder implementation, stills needs some tidying. I haven't used "Tiled" or dug that deeply into DGEngine so I'm unsure of how this will interface but if I get some time I might have a further play round. There's some useful info on DT1 tiles here. From what I gather from the Riiablo implementation the "sub-tile" stuff is handled at a higher level than decoder.

I added a quick test tool to load palettes and save PNG images. This is just for debug and should be removed before merging.

Sample outputs from act1 town:

[ghost]

Great! :+1: I didn't think someone would write one this fast! Give me some time to play with it before accepting it.

[grantramsay]

Added a DS1 decoder, thought it might be helpful to generate some "Tiled" maps from, doesn't need to be merged in. It's a rather confusing format to use... I updated the test file to try print whole maps, they look ok but if you look close some stuff is still missing / .not aligned properly.

Sample outputs:

[ghost]

I'll try and integrate it too. The alignment problems you see are probably due to the tile offsets not being applied. I remember seeing offsets somewhere when I glanced the D2 formats, so you have to shift the images.

[grantramsay]

Had another quick look at this. Went back to the doc I previously linked and decided to removed some of the quirky looking offset stuff from the Riiablo DT1 decoder implementation. Everything seems to line up much better now. Possibly there was a reason for that stuff, but this is probably a better starting point. Also added "sibling" textures, these look to be similar to the Diablo 1 "special cels" (drawn when certain other tiles are drawn).

Sample outputs:

[grantramsay]

@Lectem awesome, good to know what some of the missing/unknown information means!

[Lectem]

@Lectem awesome, good to know what some of the missing/unknown information means!

There are quite a few things that are still being researched (a few remaining tile flags, that are not necessarily used in the game files), I'd need some time to finish this. However, if you need more information on diablo2's format, I'd suggest joining the phrozen keep discord https://discordapp.com/invite/NvfftHY

[ghost]

I've been testing this pull request and have some questions:

1. Are the images stored indexed from 0 - size and then mapped using std::map<int, std::vector<int>> tilesById; to indexes in DS1 maps?
2. Are the DT1 indexes used in D2 always the ones in the tilesById map?
3. DT1 files have animated tiles. Am I correct to assume that you use a tilesById index and depending on the current time, you get a different tile for the same ID?
4. DT1 files has random tiles. Do these change whenever a level gets loaded or only once, when that level gets first loaded and then stay the same for that level/game save?
5. Are there any more of these tiles where the returned image might be different?
6. I see some of the indexes in tilesById are bigger than 32767, which means that I need to change the index type in LevelMap from int16_t to int32_t.

DS1:

1. I see this part about siblingTileId. A I correct to assume that, for instance, layer 1 indexes can be mapped to draw more than 1 layer? as in: index 0 of the floor layer can draw the floor layer and some dirt on top of it in a second layer that's right on top of the floor layer?

DGEngine can map TexturePacks to LevelMap layers. For instance, in Diablo, there's only 1 layer (equal to DUN) and that layer's indexes are used to draw all layers, including the map layer, so

• layer 1 -> TexturePack 1 (floor)
• layer 1 -> TexturePack 2 (pillars)
• layer 1 -> TexturePack 3 (pillars 2 - used in dungeon and catacombs for arches over doors)
• layer 1 -> TexturePack 4 (automap)

in Flare, we have:

• layer 1 -> TexturePack 1 (automap)
• layer 2 -> TexturePack 2 (floor)
• layer 3 -> TexturePack 2 (houses, trees, etc)
• layer 4 -> TexturePack 2 (used in snow maps as bits of snow on top of certain tiles)

What I'm trying to do is determine if I can use the current code to load DS1 files.

Let's do it one step at a time. @grantramsay, create a separate pull request with just these changes: DT1ImageConntainer, ParseImageContainer and CMakeLists.txt.

I want to change some things to better integrate DT1 support and it's easier like this. I'll then commit my changes on top of yours.

What I'm going to do:

1. Change the code to use the tilesById code in 'TexturePack's. There's a texture pack that has something similar.
2. look into the animated tiles implementation done for flare and modify it to support DT1's animated tiles and random tiles.

Afterwards, this is what I'll try to do: try and load DS1 layers into LevelMap. LevelMap uses 64x32 layers, but the ones in D2 are bigger:

DT1 tiles are similar to the big ones above, but DGEngine uses the ones in the middle for Diablo and Flare.

To load DS1 files into LevelMap, as it is right now, you would have to either:

• change the level's default tile size to be bigger
• create a LevelMap with 2x the size of a DS1 file and leave every odd tile index blank in LevelMap, so as not to overlap tiles when drawing

[grantramsay]

Quick reply from phone, might update later on: DT1:

1. I initially indexed the tiles from 0 -> size, as this seemed simple for the "Tiled" implementation. However tiles are never indexed like that in the game so could remove that and only use the proper DS1 indexing (std::map<int, std::vector<int>> tilesById;).

2. Yes (as above).

3. Yes, but could implement it however you like. E.g. the tile could keep it's animation index and just return the next frame on each call. Or you could add another index to the getter method etc, etc. Also animations are fixed to 10Hz frame rate.

4. Only once when level is loaded.

5. Not that I know of.

6. Yes you'll need at least 24bits to store the tile id.

DS1:

1. I'm not that sure about sibling textures yet. But what you've mentioned sounds likely.

Sounds good! I'll update PR later on

[grantramsay]

I'll leave this open if anyone wants to view the DS1 decoder and test files: https://github.com/grantramsay/DGEngine/pull/1

[ghost]

If the indexes used are always the ones in tilesById, then the first thing would be to modify the get to internally translate the index using tilesById.

When I was looking at the DS1 test cpp file, I saw maps being used in the draw tile. if it's not too hard, try and convert that to use a vector of size (width x height) and use -1 if that index is empty.

My idea is to construct a LevelMap object from the DS1 file and test the map using something like this:

{
  "action": {
    "name": "if.equal",
    "param1": "%game.hasPalette.level%",
    "param2": false,
    "then": {
      "name": "loadJson",
      "json": {
        "palette": {
          "id": "level",
          "fromId": "act1"
        }
      }
    }
  },
  "texturePack": [
    {
      "id": "fence",
      "file": "data/global/tiles/ACT1/TOWN/fence.dt1",
      "imageContainer": "fence",
      "palette": "level"
    },
    {
      "id": "floor",
      "file": "data/global/tiles/ACT1/TOWN/floor.dt1",
      "imageContainer": "floor",
      "palette": "level"
    },
    {
      "id": "objects",
      "file": "data/global/tiles/ACT1/TOWN/objects.dt1",
      "imageContainer": "objects",
      "palette": "level"
    },
    {
      "id": "trees",
      "file": "data/global/tiles/ACT1/TOWN/trees.dt1",
      "imageContainer": "trees",
      "palette": "level"
    }
  ],
  "level": {
    "id": "level",
    "layers": [
      { "index": 0, "texturePack": "fence" },
      { "index": 1, "texturePack": "fence" }
    ],
    "defaultLight": 255
  }
}

in the layers section, index would be a DS1 layer and "texturePack" is the DT1 files used to draw that layer

[grantramsay]

I'm still not that up to speed with DGEngines internal workings so bear with me...

If the indexes used are always the ones in tilesById, then the first thing would be to modify the get to internally translate the index using tilesById.

Why not just ditch the other indexing and rename tilesById -> tiles, you previously mentioning increasing your indexing variable size from 16 -> 32 bits?

When I was looking at the DS1 test cpp file, I saw maps being used in the draw tile. if it's not too hard, try and convert that to use a vector of size (width x height) and use -1 if that index is empty.

Can you elaborate a little more on what you're after, maybe a little example?

"texturePack": [
{
  "id": "fence",
  "file": "data/global/tiles/ACT1/TOWN/fence.dt1",
  "imageContainer": "fence",
  "palette": "level"
},

The DS1 files contain the relevant DT1 files? Why not just:

"texturePack": [
{
  "id": "TownN1",
  "file": "data/global/tiles/ACT1/TOWN/TownN1.ds1",
  "ds1ImageContainer": "TownN1",
  "palette": "level"
}],

"level": {
  "id": "level",
  "layers": [
    { "index": 0, "texturePack": "fence" },
    { "index": 1, "texturePack": "fence" }
  ],
  "defaultLight": 255
}

Can you clarify what the layers array would do, so I can better understand the implementation and make some more helpful comments

[ghost]

The JSON above is what I was using to get the map showing the floor in DGEngine (for now). TexturePacks use 32 bit indexes, but since maps don't usually address more than 32k tiles, I was using int16_t for those.

The part where I said to use tilesById inside get I did internally for now, as that's what will be used by DGEngine when drawing the level, so any offsets/id mapping should happen inside get. other TexturePacks do something similar (for Flare maps, where the indexes start at 16, and for IndexedTexturePack).

Basically, the drawTile function should call texturePack.get(idx) (in DGEngine, not in DS1_test) and get a TextureInfo struct with the offset to apply for the ID in the DS1 file for that specific tile.

struct TextureInfo
{
    const sf::Texture* texture{ nullptr };
    sf::IntRect textureRect;
    sf::Vector2f offset;
    bool absoluteOffset{ false };
    BlendMode blendMode{ BlendMode::Alpha };
    std::shared_ptr<Palette> palette;
};

This is what I'm doing now to get DS1 maps to draw just the floor (for now).

What I'm having trouble figuring out is this (ds1.floors). Is there any reason this is a map and not an array?

    // Draw floors
    for (int i = 0; i < ds1.numFloors; i++)
        drawTiles(ds1.floors, ds1.numFloors, i, { DT1::Orientation::FLOOR });

am I right to assume this is equivalent (or can be made to be equivalent) to the floor map layer:

    int32_t floorLayer[50][50];

Once I figure out how to load a layer from the DS1 class into a LevelMap layer and have it draw correctly, the others shoulkd be similar.

PS: I don't want to read DT1 files from the DS1 files. you would load them into texturePacks and load a Json that loads those in the right order.

[grantramsay]

Ahh I C, thanks for the clarification!

What I'm having trouble figuring out is this (ds1.floors). Is there any reason this is a map and not an array?

// Draw floors
for (int i = 0; i < ds1.numFloors; i++)
    drawTiles(ds1.floors, ds1.numFloors, i, { DT1::Orientation::FLOOR });

am I right to assume this is equivalent (or can be made to be equivalent) to the floor map layer:

int32_t floorLayer[50][50];

Now knowing more about DS1 files, yes those Cell maps could definitely be arrays/vectors.

ds1.floors can be made up of 0 -> MAX_FLOORS layers. So you'd need a 2D array for each layer, e.g. int32_t floors[50][50][MAX_FLOORS]; to store all the data for floors from a DS1.

A couple things to watch out for:

• additional DS1 index increment at the end of each row (see drawTiles -> index += increment;). The DS1 seems to contain an extra index at the end of each row and column, not sure what they're for but seems ok to skip them.
• importance of this line drawTile -> if (orientations.count(tile.orientation.rawValue()) != 0), every DS1 layer contain a tile for every place on the map, but only the tiles that match the orientation are correct. e.g. floor tiles in ds1.walls are simply incorrect and should be ignored...

[grantramsay]

• additional DS1 index increment at the end of each row (see drawTiles -> index += increment;). The DS1 seems to contain an extra index at the end of each row and column, not sure what they're for but seems ok to skip them.

I retested this and it seems wrong, there actually is another row/column of correct tiles... I've updated https://github.com/grantramsay/DGEngine/pull/1 to reflect this fix

[grantramsay]

Updated the test file to convert maps into a vector (height x width x layer) before drawing tiles as you suggested (https://github.com/grantramsay/DGEngine/pull/1). I've also been trying to figure out what the object/special tiles are and how to draw them, will update if/when I figure them out :/

[grantramsay]

Had a further look at DS1 objects, they are things like barrels/fires etc, often animated and interactive. They have COF/DCC files and also require an additional "obj.txt" file since the lookup table was hardcoded in original game. The DS1 files also contain NPC/monster positions and the paths they walk. Not sure if you want to use any of this additional info from the DS1s or just recreate in json files. Sample gif:

[ghost]

I might use some of that information in the future, but not for now.

I'm redoing lighting and level drawing and I'll add DS1 after that.

[ghost]

The rewrite into vectors introduced an error:

The previous one outputs correctly.

some erros I get:

• missing #include <functional>
• this code (changed):

  std::vector<std::string> dt1Filenames;
  for (const auto& filename : ds1.files)
  {
      auto newFilename = filename;
      strReplace(newFilename, ".tg1", ".dt1");
      strReplace(newFilename, "\\", "/");
      strReplace(newFilename, "/d2/data/", "data/");
      printf("%s\n", newFilename.c_str());
      dt1Filenames.push_back(newFilename);
  }

I'm almost there :)

[grantramsay]

I get that same output if I don't remove the

    width  -= 1;
    height -= 1;

from this commit I'm building on Mac, I think the std lib headers must be slightly different to other platforms as I often miss includes.

Sounds good!

[ghost]

That worked.

The drawTile function is going through all dt1 files in order to draw them. does it have to be like this? Isn't there a 1-1 mapping between a layer and a dt1 file?

        for (auto& dt1 : dt1s)
        {
            const auto& tileOptions = dt1.getTilesById(id);
            if (tileOptions.size() > 0)
            {

[grantramsay]

Annoyingly no, there can even be multiple tiles with same ID across multiple DT1s. Make sure you're looking at the latest ds1_test.cpp too as I fixed that it up a bit to account for this

[ghost]

I have it working. :+1:

Create another pull request with just DS1 and I'll commit my changes on top of yours.

I haven't done the animated tiles nor the random ones yet, but that's easy to do. The new lighting implementation isn't finished yet. Only the player's light works OK for now. And there's a new level drawing optimization (which doesn't apply to D2) for the case where all the tiles in a layer come from the same texture (I use a VertexArray in this case - much faster on both CPU and GPU).

You will have to change the test program after my changes to compile, as I changed some things.

[grantramsay]

Awesome, just opened DS1 PR https://github.com/dgengin/DGEngine/pull/31 The DS1 test stuff is still avialable here

I use a VertexArray in this case - much faster on both CPU and GPU

Nice, I tried to do something similar for freeablo in this PR. I tried putting all the game textures into an array of atlas textures, it increased the frame rate dramatically, but would be too many textures for games bigger than Diablo 1. Having a texture for each level and layer of drawing would be a good approach