tfjs-node support for saved models does not recognize valid dtypes

[vladmandic]

Simply calling tfnode.node.getMetaGraphsFromSavedModel(path); on a model using uint8 results in error:

(node:2420) UnhandledPromiseRejectionWarning: Error: Unsupported tensor DataType: DT_UINT8, try to modify the model in python to convert the datatype
    at mapTFDtypeToJSDtype (/home/vlado/dev/test-tfjs/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:465:19)

However, support for unint8 was added to tfjs via https://github.com/tensorflow/tfjs/pull/2981 back in March.
Those newly supported data types should be added throughout tfjs codebase.

Environment: Ubuntu 20.04 running NodeJS 14.9.0 with TFJS 2.3.0

[vladmandic]

More details:

There is an issue with int32 vs uint8 mapping which causes failure on execution of a saved_model in tfjs. Model in question is EfficientDet from TFHub: https://tfhub.dev/tensorflow/efficientdet/d0

I've found your an earlier fix https://github.com/tensorflow/tfjs/pull/2981 which patches tfjs-converter by mapping uint8 to int32, but:

a) Same is also needed in tfjs-node/saved_model:mapTFDtypeToJSDtype() (and possibly in other places) :

  Error: Unsupported tensor DataType: DT_UINT8, try to modify the model in python to convert the datatype
  at mapTFDtypeToJSDtype (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:471:19)

b) During model execution, model expects to receive uint8, but receives int32 and fails with:

  Error: Session fail to run with error: Expects arg[0] to be uint8 but int32 is provided
  at NodeJSKernelBackend.runSavedModel (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1592:43)  

So I'm not sure that simply mapping uint8 to int32 is a fix?

Referencing previous issue https://github.com/tensorflow/tfjs/issues/3374 closed without resolution.

Gist with a test code is at https://gist.github.com/vladmandic/a7cf75109b7b48f8914a5b18da5c498f Links for direct download of a saved_model are also included.

[vladmandic]

Thanks @pyu10055,

I've confirmed on several saved_model models that tfjs-node now works perfectly with uint8 data type. Since https://github.com/tensorflow/tfjs/pull/3974 is already committed to master, I'm closing this issue.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[loretoparisi]

More details:

There is an issue with int32 vs uint8 mapping which causes failure on execution of a saved_model in tfjs. Model in question is EfficientDet from TFHub: https://tfhub.dev/tensorflow/efficientdet/d0

I've found your an earlier fix #2981 which patches tfjs-converter by mapping uint8 to int32, but:

a) Same is also needed in tfjs-node/saved_model:mapTFDtypeToJSDtype() (and possibly in other places) :

  Error: Unsupported tensor DataType: DT_UINT8, try to modify the model in python to convert the datatype
  at mapTFDtypeToJSDtype (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:471:19)

b) During model execution, model expects to receive uint8, but receives int32 and fails with:

  Error: Session fail to run with error: Expects arg[0] to be uint8 but int32 is provided
  at NodeJSKernelBackend.runSavedModel (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1592:43)  

So I'm not sure that simply mapping uint8 to int32 is a fix?

Referencing previous issue #3374 closed without resolution.

Gist with a test code is at https://gist.github.com/vladmandic/a7cf75109b7b48f8914a5b18da5c498f Links for direct download of a saved_model are also included.

The same issue happens with DT_INT64 as well:

2020-10-02 17:55:36.860435: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-02 17:55:36.883827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x119b7b0a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-02 17:55:36.883865: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
(node:39361) UnhandledPromiseRejectionWarning: Error: Unsupported tensor DataType: DT_INT64, try to modify the model in python to convert the datatype

after converting a GraphModel to SavedModel using tensorflowjs_converter.

[vladmandic]

@loretoparisi Yup, I've reported that under https://github.com/tensorflow/tfjs/issues/4004 and it was just fixed in https://github.com/tensorflow/tfjs/pull/4008

[loretoparisi]

Hey @vladmandic in the meantime tfjs_graph_converter added a support while converting using --compat_mode in latest version 1.4.0:

$ tfjs_graph_converter --output_format tf_saved_model --compat_mode ./ ./saved/
TensorFlow.js Graph Model Converter

Graph model:    ./
Output:         ./saved/
Target format:  tf_saved_model

Converting.... Done.
Conversion took 1.667s

The GraphModel now converted to SavedModel seems loading fine now:

const tfjsnode = require('@tensorflow/tfjs-node');
var loadSavedModel = function (path) {
  return new Promise(function (resolve, reject) {
    tfjsnode.node.loadSavedModel(this.path)
      .then(res => {
        console.log("loadSavedModel OK");
        resolve(res);
      })
      .catch(err => reject(err));
  });
}
loadSavedModel('/Users/loretoparisi/webservice/toxicity_model/saved')
  .catch(err => console.error("loadSavedModel", err));

This works fine:

2020-10-05 12:38:29.166043: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /Users/loretoparisi/webservice/toxicity_model/saved
2020-10-05 12:38:29.193234: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-10-05 12:38:29.252666: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-10-05 12:38:29.252729: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /Users/loretoparisi/webservice/toxicity_model/saved/variables/variables.index
2020-10-05 12:38:29.252752: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 86709 microseconds.

BUT, if I apply this to the TFJS model wrapper here:

    ToxicityClassifier.prototype.loadModel = function () {
        return __awaiter(this, void 0, void 0, function () {
            return __generator(this, function (_a) {
                return [2, tfjsnode.node.loadSavedModel(path)];
            });
        });
    };

it will fail due to another error

(node:43549) UnhandledPromiseRejectionWarning: Error: SavedModel outputs information is not available yet.
    at TFSavedModel.get [as outputs] (/Users/loretoparisi/Documents/Projects/AI/tensorflow-node-examples/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:265:19)
    at ToxicityClassifier.<anonymous> (/Users/loretoparisi/Documents/Projects/AI/tensorflow-node-examples/toxicity-example/lib/toxicity/dist/index.js:101:35)
...

[patlevin]

@loretoparisi Do you still have the same problem? It works fine for me with int64 and int32 inputs with TFJS v2.5.0 and with int32 inputs in v2.3.0 and v2.4.0.

[loretoparisi]

@patlevin thank you, let me check, they have just released v2.5.0

[loretoparisi]

@patlevin @vladmandic So the model correctly loads with tfjs v2.5.0, but, in this example at least, there is a specific error that is

2020-10-07 20:49:15.662525: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: ./toxicity_model/saved
2020-10-07 20:49:15.702371: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-10-07 20:49:15.765097: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-10-07 20:49:15.765158: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: ./toxicity_model/saved/variables/variables.index
2020-10-07 20:49:15.765180: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 102655 microseconds.
(node:65757) UnhandledPromiseRejectionWarning: Error: SavedModel outputs information is not available yet.

I have opened a specific issue here https://github.com/tensorflow/tfjs/issues/4035

Thank you!

[patlevin]

@loretoparisi Interesting. I used the following code and it worked just fine:

const tf = require('@tensorflow/tfjs-node')

async function run() {
  const model = await tf.node.loadSavedModel('./models/toxicity_saved/')
  // both indexArray and valueArray are obtained from two preprocessed test phrases that I used to verify
  // model outputs
  const indexArray = [
    [0, 1], [0,2 ], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8],
    [1, 0], [1, 1], [1, 2], [1, 3]
  ]
  const valueArray = [215, 13, 53, 4461, 2951, 519, 1129, 7, 78, 16, 123, 20, 6]
  const indices = tf.tensor2d(indexArray, [indexArray.length, 2], 'int32')
  const values = tf.tensor1d(valueArray, 'int32')
  const modelInputs = {
    Placeholder_1: indices,
    Placeholder: values
  }
  const labels = model.predict(modelInputs)
  indices.dispose()
  values.dispose()
  outputs = []
  for (name in labels) {
   const prediction = labels[name].dataSync()
   const results = []
   for (let input = 0; input < 2; ++input) {
     const probs = prediction.slice(input * 2, input * 2 + 2)
     let match = null
     if (Math.max(probs[0], probs[1]) > 0.9) {
       match = probs[0] > probs[1]
     }
     p= probs.toString() // just to print out the numbers
     results.push({p, match})
   }
   outputs.push({label: name.split('/')[0], results})
  }
  for (x of outputs) {
    console.log(x)
  }
}

run()

The model methods outputs() and inputs() aren't implemented yet for the SavmedModel-class, but in case you need them for some reason, outputs and inputs can be obtained using the getMetaGraphsFromSavedModel() and getSignatureDefEntryFromMetaGraphInfo() functions.

[loretoparisi]

@patlevin thanks, I confirm that this way it works!

ip-192-168-178-22:toxicity-example loretoparisi$ node test.js 
node-pre-gyp info This Node instance does not support builds for N-API version 6
node-pre-gyp info This Node instance does not support builds for N-API version 6
2020-10-07 21:16:16.466776: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-07 21:16:16.494477: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11a0dfb60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-07 21:16:16.494524: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-07 21:16:16.624539: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: ./toxicity_model/saved
2020-10-07 21:16:16.647951: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-10-07 21:16:16.704225: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-10-07 21:16:16.704289: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: ./toxicity_model/saved/variables/variables.index
2020-10-07 21:16:16.704314: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 79774 microseconds.
{
  label: 'identity_attack',
  results: [
    { p: '0.9964505434036255,0.0035493909381330013', match: true },
    { p: '0.9999773502349854,0.00002267475429107435', match: true }
  ]
}
{
  label: 'insult',
  results: [
    { p: '0.013952560722827911,0.9860473871231079', match: false },
    { p: '0.9996521472930908,0.00034789409255608916', match: true }
  ]
}
{
  label: 'obscene',
  results: [
    { p: '0.997055172920227,0.002944822423160076', match: true },
    { p: '0.9999693632125854,0.00003068893784075044', match: true }
  ]
}
{
  label: 'severe_toxicity',
  results: [
    { p: '0.9999983310699463,0.0000016291029396597878', match: true },
    { p: '1,7.3735777483818765e-9', match: true }
  ]
}
{
  label: 'sexual_explicit',
  results: [
    { p: '0.9994053840637207,0.0005946253077127039', match: true },
    { p: '0.9999841451644897,0.00001583264020155184', match: true }
  ]
}
{
  label: 'threat',
  results: [
    { p: '0.9994139671325684,0.000586066220421344', match: true },
    { p: '0.9999756813049316,0.000024260229110950604', match: true }
  ]
}
{
  label: 'toxicity',
  results: [
    { p: '0.011850903742015362,0.988149106502533', match: false },
    { p: '0.999394416809082,0.0006055471021682024', match: true }
  ]
}

while, according to what you say, using the other way the error

Error: SavedModel outputs information is not available yet.
    at TFSavedModel.get [as outputs] (/Users/loretoparisi/Documents/Projects/AI/tensorflow-node-examples/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:251:19)

comes from within the saved_model class, in fact there we have a not implemented error!

Object.defineProperty(TFSavedModel.prototype, "outputs", {
        /**
         * Return the array of output tensor info.
         *
         * @doc {heading: 'Models', subheading: 'SavedModel'}
         */
        get: function () {
            throw new Error('SavedModel outputs information is not available yet.');
        },
        enumerable: true,
        configurable: true
    });

hence the offending line in the toxicity example is the following I assume

this.labels =
                            model.outputs.map(function (d) { return d.name.split('/')[0]; });
                        if (this.toxicityLabels.length === 0) {
                            this.toxicityLabels = this.labels;
                        }

that must be changed in someway using the getMetaGraphsFromSavedModel method then.

[patlevin]

@loretoparisi I'll create a pull-request that implements outputs() and inputs() on SavedModel.

[loretoparisi]

@loretoparisi I'll create a pull-request that implements outputs() and inputs() on SavedModel.

super! In the meantime I have found the outputs something as you suggested

const modelInfo = await tf.node.getMetaGraphsFromSavedModel('./toxicity_model/saved');
console.dir(modelInfo[0].signatureDefs.serving_default.outputs, { depth: null, maxArrayLength: null });

[vladmandic]

@loretoparisi btw, one advantage of working with saved_model and getMetaGraphsFromSavedModel() is that it shows actual signature names instead just an incrementing array (when model has multiple inputs and/or outputs) that you get from a graph_model.

See https://github.com/tensorflow/tfjs/issues/3942 for details.

[loretoparisi]

@loretoparisi Interesting. I used the following code and it worked just fine:

const tf = require('@tensorflow/tfjs-node')

async function run() {
  const model = await tf.node.loadSavedModel('./models/toxicity_saved/')
  // both indexArray and valueArray are obtained from two preprocessed test phrases that I used to verify
  // model outputs
  const indexArray = [
    [0, 1], [0,2 ], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8],
    [1, 0], [1, 1], [1, 2], [1, 3]
  ]
  const valueArray = [215, 13, 53, 4461, 2951, 519, 1129, 7, 78, 16, 123, 20, 6]
  const indices = tf.tensor2d(indexArray, [indexArray.length, 2], 'int32')
  const values = tf.tensor1d(valueArray, 'int32')
  const modelInputs = {
    Placeholder_1: indices,
    Placeholder: values
  }
  const labels = model.predict(modelInputs)
  indices.dispose()
  values.dispose()
  outputs = []
  for (name in labels) {
   const prediction = labels[name].dataSync()
   const results = []
   for (let input = 0; input < 2; ++input) {
     const probs = prediction.slice(input * 2, input * 2 + 2)
     let match = null
     if (Math.max(probs[0], probs[1]) > 0.9) {
       match = probs[0] > probs[1]
     }
     p= probs.toString() // just to print out the numbers
     results.push({p, match})
   }
   outputs.push({label: name.split('/')[0], results})
  }
  for (x of outputs) {
    console.log(x)
  }
}

run()

The model methods outputs() and inputs() aren't implemented yet for the SavmedModel-class, but in case you need them for some reason, outputs and inputs can be obtained using the getMetaGraphsFromSavedModel() and getSignatureDefEntryFromMetaGraphInfo() functions.

@patlevin thanks for the test script. I modified it a little bit to user the Universal Sentence Encoder:

const use = require("@tensorflow-models/universal-sentence-encoder");
const model = await tf.node.loadSavedModel('./toxicity_model/saved');
const tokenizer = await use.load();
const sentences = ['you suck', 'hello how are you?'];
var codes = await tokenizer.embed(sentences);
console.log(codes);

Now I have an array of Tensors like

Tensor {
  kept: false,
  isDisposedInternal: false,
  shape: [ 1, 512 ],
  dtype: 'float32',
  size: 512,
  strides: [ 512 ],
  dataId: {},
  id: 837,
  rankType: '2',
  scopeId: 1289
}

and now I have to turn into indexes and values. I have tried this

var encodings = await tokenizer.embed(sentences);
var indicesArr = encodings.map(function (arr, i) { return arr.map(function (d, index) { return [i, index]; }); });
var flattenedIndicesArr = [];
for (i = 0; i < indicesArr.length; i++) {
    flattenedIndicesArr =
        flattenedIndicesArr.concat(indicesArr[i]);
}
var indices = tf.tensor2d(flattenedIndicesArr, [flattenedIndicesArr.length, 2], 'int32');
var values = tf.tensor1d(tf.util.flatten(encodings), 'int32');

But it seems that embed function of the Tokenizers is different from the encode function internally used that I cannot call external (it gives me an undefined if I try tokenizer.encode), so in this case I get an error:

TypeError: encodings.map is not a function

Thank you.

[patlevin]

@loretoparisi The model expects an encoded word vector as input, while the Universal Sentence Encoder (USE) model returns embeddings.

Basically, you'll want to use the loadTokenizer() function from the previous USE version, but that one requires TFJS 1.x... I have a working version locally, but it'd be better to fix the examples instead - see Issue model repo

Unfortunately @pyu10055's commit b02310745ceac6b8e4a475719c343da53e3cade2 on the USE-repo broke both the Toxicity example model and your use-case entirely...

The real problem is that the examples are outdated and some changes broke TFJS 2.x compatibility (in the case of USE I fail to see the reasoning behind the change - might have been a mistake?).

Meanwhile, I'll create a gist for you that contains all you need to get this working as a single-file solution. I'll get back to you in a bit.

EDIT: I got confused here, since a similar issue was raised w.r.t. outdated tfjs-examples. The same applies to tfjs-models, though - basically some models are incompatible with TFJS 2.x due to package changes (not for technical reasons).

[loretoparisi]

@patlevin thank you! In fact I have just realized that the after recent updates models examples and core code diverged!

[playground]

I have the same issue, running node v15.7.0, tfjs-node v3.6.1 on Mac Big Sur

  const image = fs.readFileSync(imagePath);
  const decodedImage = tfnode.node.decodeImage(new Uint8Array(image), 3);
  const model = await tfnode.node.loadSavedModel(modelPath);
  const predictions = model.predict(decodedImage);

2021-05-31 18:58:58.942711: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: saved_model
2021-05-31 18:58:59.291324: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-05-31 18:58:59.291368: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: saved_model
2021-05-31 18:59:00.692648: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-05-31 18:59:03.160455: I tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: saved_model
2021-05-31 18:59:04.360189: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 5417477 microseconds.
2021-05-31 18:59:12.511652: E tensorflow/core/framework/tensor.cc:555] Could not decode variant with type_name: "tensorflow::TensorList".  Perhaps you forgot to register a decoder via REGISTER_UNARY_VARIANT_DECODE_FUNCTION?
2021-05-31 18:59:12.511706: W tensorflow/core/framework/op_kernel.cc:1740] OP_REQUIRES failed at constant_op.cc:79 : Invalid argument: Cannot parse tensor from tensor_proto.
2021-05-31 18:59:12.546008: E tensorflow/core/framework/tensor.cc:555] Could not decode variant with type_name: "tensorflow::TensorList".  Perhaps you forgot to register a decoder via REGISTER_UNARY_VARIANT_DECODE_FUNCTION?
2021-05-31 18:59:12.546526: W tensorflow/core/framework/op_kernel.cc:1740] OP_REQUIRES failed at constant_op.cc:79 : Invalid argument: Cannot parse tensor from proto: dtype: DT_VARIANT
tensor_shape {
}
variant_val {
  type_name: "tensorflow::TensorList"
  metadata: "\001\000\001\377\377\377\377\377\377\377\377\377\001\030\001"
}

/test-saved-model/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:449
        var outputMetadata = this.binding.runSavedModel(id, this.getMappedInputTensorIds(inputs, inputTensorInfos), inputTensorInfos.map(function (info) { return info.name; }).join(','), outputOpNames.join(','));
                                          ^

Error: Session fail to run with error: Cannot parse tensor from proto: dtype: DT_VARIANT
tensor_shape {
}
variant_val {
  type_name: "tensorflow::TensorList"
  metadata: "\001\000\001\377\377\377\377\377\377\377\377\377\001\030\001"
}