Better serialization support

[GoogleCodeExporter]

Can we get better save/load functionality?

It seems simple enough to save the mesh, something like this

File f;
if(f.OpenForWrite(navPath.c_str(),File::Binary))
{
    f.WriteInt32(DetourNavmesh.getMaxTiles());
    for(int t = 0; t < DetourNavmesh.getMaxTiles(); ++t)
    {
        f.WriteInt32(tile->dataSize);
        const dtMeshTile* tile = DetourNavmesh.getTile(t);
        f.Write(tile->data,tile->dataSize);
    }               
    f.Close();
    return true;
}

but as I load the mesh again, I seem to be missing information that it
wasn't clear needed saving explicitly, such as most of the parameters
passed to the longer version of dtNavMesh::Init.

It appears that there is really only consideration right now for saving the
individual tiles, and not a save/load interface for saving the navmesh
properties themselves.

Also, in saving the tiles in this way, information about where the tile is
is lost, which makes it more difficult at load time to re-add the tiles
since dtNavMesh::addTileAt requires the x,y.

Request
-------
Can we get more straightforward functionality for serializing the navmesh
and tiles? I like that tiles have a data and data size pointer that makes
saving them easy, but in order to reconstruct and load the navmesh again
requires initialization information for the navmesh, and tile locations to
re-add the tiles. 

It would seem to me that the tile x,y locations (passed to addTile)
locations should be save-able with the tile data, or even better the tile
locations could be re-calculated at add-time based on the bounds of the tile.

Thoughts?

Original issue reported on code.google.com by jswig...@gmail.com on 6 Dec 2009 at 1:17

[GoogleCodeExporter]

It is also important to save the tile salt, so that the tile refs stay valid 
across
serialization.

I always thought that depending on how the tiles are used (streaming, dynamic
generation, etc) there would be higher level system which keeps track of which 
tiles
are in and out. That system would be also responsible for serialization.

There is some info lacking there. I'll add the missing serialization stuff 
there and
perhaps write an example on how I expect the serialization to be used.

Original comment by memono...@gmail.com on 6 Dec 2009 at 8:34

• Changed state: Accepted

[GoogleCodeExporter]

That's fair. I don't mind handling which tiles to save/load, but it'd be nice 
to save
the core navmesh settings so that the navmesh can be initialized on load and 
then I
can start loading in tiles immediately.

The navmesh options, like tile size and such, are effectively a constant after 
you
build a navmesh aren't they(as in, there's no support for resizing the tile 
size and
re-using tiles?)? Also isn't the tile data in world space? If so, couldn't 
adding the
tile data handle its own placement and validation against the tile size? so 
addTile
could just take the data * and size?

Original comment by jswig...@gmail.com on 6 Dec 2009 at 5:25

[GoogleCodeExporter]

I went through the code and tried to retrace my thoughts. I think at the time of
writing the code I was thinking about the ability to move the tiles around in 
order
to keep the data as close to center as possible. I think I was also considering 
to
have the tile data to be relative to the tile origin. This did not happen, I 
need to
revisit the idea to see how complicated it might get to make it happen. It 
might be a
good idea.

Otherwise the location of the tile could be encoded in the tile data itself.

I think it would not be too much trouble to have the inner workings of the 
navmesh to
be exposed ans nice chunk of data which can be serialized nicely. I'm not sure 
how to
add the required tiles later, though.

I theory it could go like this:

To write:
ser.write(navmesh.getStateData(), navmesh.getStateDataSize());
for (int i = 0; i < navmesh.getTileCount(); ++i)
{
    if (navmesh.hasTile(i))
    {
        ser.write(navmesh.getTileData(i), navmesh.getTileDataSize(i));
    }
}

To read:
ser.read(stateData, stateDataSize);
navmesh.restore(statedata, stateDataSize);
for (int i = 0; i < navmesh.getTileCount(); ++i)
{
    if (navmesh.hasTile(i))
    {
        ser.read(tileData, tilDataSize);
        navmesh.restoreTile(i, tileData,tileDataSize);
    }
}

I don't completely like it, but it would be better than it is currently. I 
think the
complications come from the fact that in certain use cases you would only store 
the
navmesh state, not the data for all the times as they may be static and read 
from
disk. Something like this:

ser.read(stateData, stateDataSize);
navmesh.restore(statedata, stateDataSize);
for (int i = 0; i < navmesh.getTileCount(); ++i)
{
    if (navmesh.hasTile(i))
    {
        resourceManager.getNavmeshTileData(navmesh.getTileUserId(i), tileData, tileDataSize);
        navmesh.restoreTile(i, tileData,tileDataSize);
    }
}

That way each tile could have associated user id, which allows to recognize 
which
data needs to be loaded.

The read back method above is a bit too loose to my liking. That is, it is a 
bit too
easy to break the data.

Another way to do this would be to require to pass a callback function to the
navmesh.restore(), but I dont quite like that either.

I will think about another solution to this still. If I cannot come up with 
anything
better, I will try the above method. It is an improvement.

If you have any thoughts so far, let me know.

Original comment by memono...@gmail.com on 9 Dec 2009 at 10:39

[GoogleCodeExporter]

Sounds pretty good. Would the tile data not be stored in the navmesh? I would 
have
thought the navmesh data itself would have a big array of tile data, rather than
needing to save/load each tile data in separately. 

If I were making a tile/streaming game, I probably wouldn't save like the 
above, I
would save each tile to a separate file, ie tile_<tileid>.nm, and at load time I
would load the core navmesh data, and then let the streaming system or the
locationally aware loading logic add bubbles of relevancy that would load only 
select
tiles. I would imagine the navmesh to have the tile data stored within itself, 
so
that I can query just the navmesh(without any tiles loaded) for tiles touching 
that
relevancy sphere or whatever and using the tile ids returned, start loading the
actual tile data.

Original comment by jswig...@gmail.com on 9 Dec 2009 at 2:36

[GoogleCodeExporter]

I think I agree. Currently the dtNavMesh (which is based on the old 
dtTileMesh), has
only very little internal state. In practice the only important data that needs 
to be
serialized is the per tile salts. That ensures that dtPolyRefs wil be valid 
after
serialization.

I would store the navmesh pieces the same way as you would. I would think each 
tile
as a resource (much like texture or what ever). The second example with
resourceManager.getNavmeshTileData() et al was sort of describing that sort of 
tile
configuration restoration step.

I think it can be that we are talking slightly different usage cases here. I was
thinking more about the case that you want to store the navmesh tile 
configuration at
runtime, say into a save game. Another use case could be to store all the 
settings of
the navmesh. Sort of like empty initial state which has all the tile sizes etc 
set up
the same way as when the tiles were generated.

I think the same code can be used for both.

There are certain things here I want to avoid to make the system flexible. The
example with saving the tile navmeshes was sort of trying to solve the 
situation that
you have completely dynamic environment and you really want to save all the 
data.

Also, I think there is no single good way to handle the overall layout of the 
tiles.
I think that is partially the solution you are looking for. What I am implying 
here
is that the manager which decides which tiles to load will be different for 
every
game, the navmesh just has support to remove and add tiles to it and it handles 
the
nasty case of patching the boundaries.

Most people are going to wrap Detour when using it in their projects and I have 
tried
to create Detour is such way that it is possible, which means that some 
features are
not "fully" implemented. Then I try to provide ideas how to create such features
using the samples.

I think I'll approach this issue by making two examples: One is dynamic mesh use
case, and another is simple semi-fake streaming example. Both should also 
support
saving the current state of the navigation mesh tile configuration. I'm sure I 
will
do less guessing that way too :)

Original comment by memono...@gmail.com on 9 Dec 2009 at 3:34

[GoogleCodeExporter]

#define MAX_NODES 2048

struct navMeshSettings
{
    float orig[3];
    float tileWidth;
    float tileHeight;
    float portalHeight;
    int maxTiles;
    int maxPolys;
};

bool OpenNavMesh(char *name)
{
    CreateDirectory("nav", 0);
    char buf[1024];
    sprintf(buf, "nav\\%s.nms", name);
    FILE *fp=fopen(buf, "rb"); 
    if (!fp)
        return false;
    fseek(fp, 0, 2);
    int size = ftell(fp);
    fseek(fp, 0, 0);
    fread(&m_navMeshSettings, size, 1, fp);
    m_navMesh = new dtNavMesh;
    m_navMesh->init(m_navMeshSettings.orig, m_navMeshSettings.tileWidth, 
m_navMeshSettings.tileHeight, m_navMeshSettings.portalHeight, 
m_navMeshSettings.maxTiles, m_navMeshSettings.maxPolys, MAX_NODES);
    printf("Loaded %s nav mesh settings, %d bytes\r\n", name, size);

    sprintf(buf, "nav\\%s.nav", name);
    FILE *fp=fopen(buf, "rb");
    if (!fp)
        return false;
    fseek(fp, 0, 2);
    size = ftell(fp);
    fseek(fp, 0, 0);
    unsigned char *data = new unsigned char [size];
    fread(data, size, 1, fp);

    // Remove any previous data (navmesh owns and deletes the data).
    m_navMesh->removeTileAt(current_x,current_y,0,0);
    // Let the navmesh own the data.
    if (!m_navMesh->addTileAt(current_x,current_y,data,size,true))
        delete [] data;
    return true;
}

bool SaveNavMesh(char *name, unsigned char *data, int size)
{
    CreateDirectory("nav", 0); //win32api call to make folder for nav meshes
    char buf[1024];
    sprintf(buf, "nav\\%s", name);
    FILE *fp=fopen(buf, "wb"); //without it you cant fopen in nonexistant folders
    if (!fp)
        return false;
    fwrite(data, size, 1, fp);
    fclose(fp);
    printf("Saved %s nav mesh, %d bytes\r\n", name, size);
    return true;
}

Usage:
unsigned char* data = buildTileMesh(m_tileBmin, m_tileBmax, dataSize);
SaveNavMesh(name, data, dataSize);

Original comment by kork...@gmail.com on 20 Dec 2009 at 1:48

[GoogleCodeExporter]

I can see one big thing missing. As much as I love recast's feature to have all 
the
data packed up in nice, consistent blocks, and as such very easy to dump to a 
file
and restore, this doesn't help at all if you're compiling your resources for
consoles. For those unfamiliar with the topic, consoles (XBox360, PS3) use
big-endian, PCs little-endian. So if on my PC I'm preparing my navmesh to be 
used on
a console I can't simply dump the memory into a file. Also I can't naively swap 
bytes
in the memory, since navmesh data doesn't consist of a single data type.

So, my biggest concern is not when to load tiles, how to attach them, and what 
about
nav mesh config data. My biggest problem is to serialize navmesh/tiles in PC's 
memory
with my console-friendly serializer in a way that will allow me to read it on a
console just as if it would be a mem dump from a console (sorry is it's a little
unclear). I managed to do that with recast 1.2, but after integrating 1.4.1 I'm 
a
little confused.

My ideal solution would be to have Recast specify an interface it will use to
serialize navmesh. Something like:

class INavmeshSerializer
{
    virtual bool Serialize(float&) = 0;
    virtual bool Serialize(int&) = 0;
    virtual bool Serialize(char&) = 0;
    ...
    virtual int SerializeArray(float* data, int size) = 0;
    virtual int SerializeArray(int* data, int size) = 0;
    virtual int SerializeArray(char* data, int size) = 0;
    ...
    // etc.
};

user would create a class implementing this interface and simply pass it as a 
navmesh
serialization call, like this:

class MySerializer : public INavmeshSerializer
{
    // all required virtuals implemented...
}

// and when navmesh is to be serialized:
MySerializer mySerializerInstance("filename.nm"); // or some data stream
pNavmesh->Serialize(mySerializerInstance);

the same could go for loading since serialized would know if it's storing or
restoring. But this solution makes most sense for saving data.

One of the benefits of this approach would be further encapsulation of navmesh
internals from pesky programmers, and internal data structure changes could be
transparently handled without them even knowing ;)

So... does this make sense? How hard would it be to do?

Original comment by mieszko....@gmail.com on 30 Dec 2009 at 1:02

[GoogleCodeExporter]

Makes perfect sense to me and sounds great. Would there be separate calls to 
load the core navmesh 
data and load individual tiles with different calls to facilitate 
streaming/partial loading?

Original comment by jswig...@gmail.com on 30 Dec 2009 at 6:49

[GoogleCodeExporter]

Original comment by memono...@gmail.com on 12 Feb 2010 at 2:52

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

this reminds me, with the recent addition of the off mesh connections, it 
appears
that we need to save our own off mesh information separate from the navmesh, is 
that
correct? some stuff can presumably be derived from game constructs like ladders 
and
teleports and stuff, but jumps are often manually placed, and it's slightly more
complexity to manage that data outside of the navmesh

Original comment by jswig...@gmail.com on 12 Feb 2010 at 5:22

[GoogleCodeExporter]

I arranged stuff a bit differently and added helper functions in R147. If more 
work
is needed, I will create new issues as they appear.

Original comment by memono...@gmail.com on 26 Mar 2010 at 2:07

• Changed state: Fixed