d = tds.Hash([tbl])

Creates a hash table which implements the lua operators [key], # and pairs(), and in very similar way than lua tables.

A hash can contain any element (either as key or value) supported by tds.

d = tds.Vec([... || tbl])

Creates a vector of elements indexed by numbers starting from 1. If a single lua table tbl (or several arguments) is (are) passed at construction, the vector will be filled with the lua table contents (or the given arguments).

d:concat([sep, i, j])

Concat all vector elements into a single string. Fails if an element cannot be converted via tostring().

sep is an optional separator string inserted between each elements.

i and j define an optional range (by default i=1 and j is the size of the vector).

d:concatstorage([sep, i, j])

Serialization

All tds data structures support torch serialization. Example:

tds = require 'tds'
require 'torch'

-- create a vector containing heterogeneous data
d = tds.Vec(4, 5, torch.rand(3), nil, "hello world")

-- serialize in a buffer
f = torch.MemoryFile("rw")
f:writeObject(d)

-- unserialize
f:seek(1)
print(f:readObject())

The example will output:

tds.Vec[5]{
    1 : 4
    2 : 5
    3 :  0.1665
         0.8750
         0.7525
        [torch.DoubleTensor of size 3]

    4 : nil
    5 : hello world
}

Nesting

Nesting is supported in tds. However, reference loops are prohibited, and will lead to leaks if used.

Example:

tds = require 'tds'
require 'torch'

-- create a vector containing heterogeneous data
d = tds.Vec(4, 5, torch.rand(3), tds.Hash(), "hello world")

-- fill up the hash table:
d[4].foo = "bar"
d[4][6] = torch.rand(3)
d[4].stuff = tds.Vec("how", "are", "you", "doing")

print(d)

This example will output:

tds.Vec[5]{
    1 : 4
    2 : 5
    3 :  0.1958
         0.5663
         0.2777
        [torch.DoubleTensor of size 3]

    4 : tds.Hash[3]{
        foo   : bar
        6     :  0.0105
                 0.7496
                 0.5241
                [torch.DoubleTensor of size 3]

        stuff : tds.Vec[4]{
                    1 : how
                    2 : are
                    3 : you
                    4 : doing
                }
        }
    5 : hello world
}

Extending to other C types

tds provides a way to extend to your own C types using the submodule tds.elem:

local elem = require 'tds.elem'

elem.type(obj)

tds typechecking is achieved using this function. You can override it for your own purposes. If torch is detected, tds will set elem.type to torch.typename(), so in general (if you are using torch!) you should not worry about this part.

elem.addctype(ttype, free_p, setfunc, getfunc)

Add a new C type into tds:

• ttype must be the typename understood by the current elem.type() function.
• free_p is a C FFI pointer to a destructor of the C object.
• setfunc(luaobj) takes a lua object and returns a FFI C pointer on this object, as well as a FFI function free_p to free this object.
• getfunc(cpointer) takes a C FFI pointer and returns a lua object of the corresponding object.

One must be careful to handle properly reference counting and garbage collection in setfunc() and getfunc():

• setfunc() will convert a lua object into a C pointer which will be stored into the data structure: the reference count on this object must be increased. When removed from the data structure, tds will call the given free_p() function.
• getfunc() will convert a C pointer and push it into lua memory space: one must again increase properly the reference count on this object, and make sure lua will garbage collect it properly.

Here is a typical example showing how support for tds.Hash elements is supported:

   elem.addctype(
      'tds.Hash',
      C.tds_hash_free,
      function(lelem)
         C.tds_hash_retain(lelem)
         return lelem,  C.tds_hash_free
      end,
      function(lelem_p)
         local lelem = ffi.cast('tds_hash&', lelem_p)
         C.tds_hash_retain(lelem)
         ffi.gc(lelem, C.tds_hash_free)
         return lelem
      end
   )