search

yorkie / tensorflow-nodejs

TensorFlow Node.js provides idiomatic JavaScript language bindings and a high layer API for Node.js users. TensorFlow Node.js provides idiomatic JavaScript language bindings and a high layer API for Node.js users.
https://npmjs.org/package/tensorflow2
MIT License
579 stars 88 forks source link

tensor: support string encode/decode to a tensor #1

Closed yorkie closed 7 years ago

yorkie commented 7 years ago

The API might be here: https://github.com/yorkie/tensorflow-nodejs/blob/master/tensorflow/include/tensorflow/c/c_api.h#L275, and the answer might be TF_StringEncode and TF_StringDecode :)

yorkie commented 7 years ago

Interesting working with TF_StringEncode and TF_StringDecode function, also find the calls to TF_StringEncodedSize is 2 for normal usage. Anyway, it works by this commit 74e2426.

yorkie commented 7 years ago

Oh 74e2426 just are the partial of this issue, reopening this.

The full design of this string tensor is: a string tensor is an array of strings, and every elements are encoded with the function TF_StringEncode, and the whole buffer structure are: elements offsets + elements buffer.

For example, if we want to encode a string: [ "foobar" ], it's:

offsets[1] + [ encode("foobar") ]

and [ "foobar", "beep" ]:

offsets[1, x] + [ encode("foobar"), encode("beep") ]

In offsets part, it's an array, too. every element is an uint64 to represent the offset of its elements in the whole buffer. That means the offsets length is the elements number.