Closed vladmandic closed 4 years ago
More details:
There is an issue with int32
vs uint8
mapping which causes failure on execution of a saved_model
in tfjs.
Model in question is EfficientDet from TFHub: https://tfhub.dev/tensorflow/efficientdet/d0
I've found your an earlier fix https://github.com/tensorflow/tfjs/pull/2981 which patches tfjs-converter
by mapping uint8
to int32
, but:
a) Same is also needed in tfjs-node/saved_model:mapTFDtypeToJSDtype()
(and possibly in other places) :
Error: Unsupported tensor DataType: DT_UINT8, try to modify the model in python to convert the datatype
at mapTFDtypeToJSDtype (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:471:19)
b) During model execution, model expects to receive uint8
, but receives int32
and fails with:
Error: Session fail to run with error: Expects arg[0] to be uint8 but int32 is provided
at NodeJSKernelBackend.runSavedModel (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1592:43)
So I'm not sure that simply mapping uint8 to int32 is a fix?
Referencing previous issue https://github.com/tensorflow/tfjs/issues/3374 closed without resolution.
Gist with a test code is at https://gist.github.com/vladmandic/a7cf75109b7b48f8914a5b18da5c498f
Links for direct download of a saved_model
are also included.
Thanks @pyu10055,
I've confirmed on several saved_model
models that tfjs-node
now works perfectly with uint8
data type.
Since https://github.com/tensorflow/tfjs/pull/3974 is already committed to master, I'm closing this issue.
More details:
There is an issue with
int32
vsuint8
mapping which causes failure on execution of asaved_model
in tfjs. Model in question is EfficientDet from TFHub: https://tfhub.dev/tensorflow/efficientdet/d0I've found your an earlier fix #2981 which patches
tfjs-converter
by mappinguint8
toint32
, but:a) Same is also needed in
tfjs-node/saved_model:mapTFDtypeToJSDtype()
(and possibly in other places) :Error: Unsupported tensor DataType: DT_UINT8, try to modify the model in python to convert the datatype at mapTFDtypeToJSDtype (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:471:19)
b) During model execution, model expects to receive
uint8
, but receivesint32
and fails with:Error: Session fail to run with error: Expects arg[0] to be uint8 but int32 is provided at NodeJSKernelBackend.runSavedModel (/home/vlado/dev/efficientdet/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1592:43)
So I'm not sure that simply mapping uint8 to int32 is a fix?
Referencing previous issue #3374 closed without resolution.
Gist with a test code is at https://gist.github.com/vladmandic/a7cf75109b7b48f8914a5b18da5c498f Links for direct download of a
saved_model
are also included.
The same issue happens with DT_INT64
as well:
2020-10-02 17:55:36.860435: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-02 17:55:36.883827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x119b7b0a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-02 17:55:36.883865: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
(node:39361) UnhandledPromiseRejectionWarning: Error: Unsupported tensor DataType: DT_INT64, try to modify the model in python to convert the datatype
after converting a GraphModel to SavedModel using tensorflowjs_converter
.
@loretoparisi Yup, I've reported that under https://github.com/tensorflow/tfjs/issues/4004 and it was just fixed in https://github.com/tensorflow/tfjs/pull/4008
Hey @vladmandic in the meantime tfjs_graph_converter
added a support while converting using --compat_mode
in latest version 1.4.0
:
$ tfjs_graph_converter --output_format tf_saved_model --compat_mode ./ ./saved/
TensorFlow.js Graph Model Converter
Graph model: ./
Output: ./saved/
Target format: tf_saved_model
Converting.... Done.
Conversion took 1.667s
The GraphModel
now converted to SavedModel
seems loading fine now:
const tfjsnode = require('@tensorflow/tfjs-node');
var loadSavedModel = function (path) {
return new Promise(function (resolve, reject) {
tfjsnode.node.loadSavedModel(this.path)
.then(res => {
console.log("loadSavedModel OK");
resolve(res);
})
.catch(err => reject(err));
});
}
loadSavedModel('/Users/loretoparisi/webservice/toxicity_model/saved')
.catch(err => console.error("loadSavedModel", err));
This works fine:
2020-10-05 12:38:29.166043: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /Users/loretoparisi/webservice/toxicity_model/saved
2020-10-05 12:38:29.193234: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-10-05 12:38:29.252666: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-10-05 12:38:29.252729: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /Users/loretoparisi/webservice/toxicity_model/saved/variables/variables.index
2020-10-05 12:38:29.252752: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 86709 microseconds.
BUT, if I apply this to the TFJS model wrapper here:
ToxicityClassifier.prototype.loadModel = function () {
return __awaiter(this, void 0, void 0, function () {
return __generator(this, function (_a) {
return [2, tfjsnode.node.loadSavedModel(path)];
});
});
};
it will fail due to another error
(node:43549) UnhandledPromiseRejectionWarning: Error: SavedModel outputs information is not available yet.
at TFSavedModel.get [as outputs] (/Users/loretoparisi/Documents/Projects/AI/tensorflow-node-examples/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:265:19)
at ToxicityClassifier.<anonymous> (/Users/loretoparisi/Documents/Projects/AI/tensorflow-node-examples/toxicity-example/lib/toxicity/dist/index.js:101:35)
...
@loretoparisi Do you still have the same problem? It works fine for me with int64 and int32 inputs with TFJS v2.5.0 and with int32 inputs in v2.3.0 and v2.4.0.
@patlevin thank you, let me check, they have just released v2.5.0
@patlevin @vladmandic So the model correctly loads with tfjs v2.5.0
, but, in this example at least, there is a specific error that is
2020-10-07 20:49:15.662525: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: ./toxicity_model/saved
2020-10-07 20:49:15.702371: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-10-07 20:49:15.765097: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-10-07 20:49:15.765158: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: ./toxicity_model/saved/variables/variables.index
2020-10-07 20:49:15.765180: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 102655 microseconds.
(node:65757) UnhandledPromiseRejectionWarning: Error: SavedModel outputs information is not available yet.
I have opened a specific issue here https://github.com/tensorflow/tfjs/issues/4035
Thank you!
@loretoparisi Interesting. I used the following code and it worked just fine:
const tf = require('@tensorflow/tfjs-node')
async function run() {
const model = await tf.node.loadSavedModel('./models/toxicity_saved/')
// both indexArray and valueArray are obtained from two preprocessed test phrases that I used to verify
// model outputs
const indexArray = [
[0, 1], [0,2 ], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8],
[1, 0], [1, 1], [1, 2], [1, 3]
]
const valueArray = [215, 13, 53, 4461, 2951, 519, 1129, 7, 78, 16, 123, 20, 6]
const indices = tf.tensor2d(indexArray, [indexArray.length, 2], 'int32')
const values = tf.tensor1d(valueArray, 'int32')
const modelInputs = {
Placeholder_1: indices,
Placeholder: values
}
const labels = model.predict(modelInputs)
indices.dispose()
values.dispose()
outputs = []
for (name in labels) {
const prediction = labels[name].dataSync()
const results = []
for (let input = 0; input < 2; ++input) {
const probs = prediction.slice(input * 2, input * 2 + 2)
let match = null
if (Math.max(probs[0], probs[1]) > 0.9) {
match = probs[0] > probs[1]
}
p= probs.toString() // just to print out the numbers
results.push({p, match})
}
outputs.push({label: name.split('/')[0], results})
}
for (x of outputs) {
console.log(x)
}
}
run()
The model methods outputs()
and inputs()
aren't implemented yet for the SavmedModel
-class, but in case you need them for some reason, outputs and inputs can be obtained using the getMetaGraphsFromSavedModel()
and getSignatureDefEntryFromMetaGraphInfo()
functions.
@patlevin thanks, I confirm that this way it works!
ip-192-168-178-22:toxicity-example loretoparisi$ node test.js
node-pre-gyp info This Node instance does not support builds for N-API version 6
node-pre-gyp info This Node instance does not support builds for N-API version 6
2020-10-07 21:16:16.466776: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-07 21:16:16.494477: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11a0dfb60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-07 21:16:16.494524: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-10-07 21:16:16.624539: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: ./toxicity_model/saved
2020-10-07 21:16:16.647951: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-10-07 21:16:16.704225: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-10-07 21:16:16.704289: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: ./toxicity_model/saved/variables/variables.index
2020-10-07 21:16:16.704314: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 79774 microseconds.
{
label: 'identity_attack',
results: [
{ p: '0.9964505434036255,0.0035493909381330013', match: true },
{ p: '0.9999773502349854,0.00002267475429107435', match: true }
]
}
{
label: 'insult',
results: [
{ p: '0.013952560722827911,0.9860473871231079', match: false },
{ p: '0.9996521472930908,0.00034789409255608916', match: true }
]
}
{
label: 'obscene',
results: [
{ p: '0.997055172920227,0.002944822423160076', match: true },
{ p: '0.9999693632125854,0.00003068893784075044', match: true }
]
}
{
label: 'severe_toxicity',
results: [
{ p: '0.9999983310699463,0.0000016291029396597878', match: true },
{ p: '1,7.3735777483818765e-9', match: true }
]
}
{
label: 'sexual_explicit',
results: [
{ p: '0.9994053840637207,0.0005946253077127039', match: true },
{ p: '0.9999841451644897,0.00001583264020155184', match: true }
]
}
{
label: 'threat',
results: [
{ p: '0.9994139671325684,0.000586066220421344', match: true },
{ p: '0.9999756813049316,0.000024260229110950604', match: true }
]
}
{
label: 'toxicity',
results: [
{ p: '0.011850903742015362,0.988149106502533', match: false },
{ p: '0.999394416809082,0.0006055471021682024', match: true }
]
}
while, according to what you say, using the other way the error
Error: SavedModel outputs information is not available yet.
at TFSavedModel.get [as outputs] (/Users/loretoparisi/Documents/Projects/AI/tensorflow-node-examples/node_modules/@tensorflow/tfjs-node/dist/saved_model.js:251:19)
comes from within the saved_model
class, in fact there we have a not implemented error!
Object.defineProperty(TFSavedModel.prototype, "outputs", {
/**
* Return the array of output tensor info.
*
* @doc {heading: 'Models', subheading: 'SavedModel'}
*/
get: function () {
throw new Error('SavedModel outputs information is not available yet.');
},
enumerable: true,
configurable: true
});
hence the offending line in the toxicity
example is the following I assume
this.labels =
model.outputs.map(function (d) { return d.name.split('/')[0]; });
if (this.toxicityLabels.length === 0) {
this.toxicityLabels = this.labels;
}
that must be changed in someway using the getMetaGraphsFromSavedModel
method then.
@loretoparisi I'll create a pull-request that implements outputs()
and inputs()
on SavedModel
.
@loretoparisi I'll create a pull-request that implements
outputs()
andinputs()
onSavedModel
.
super! In the meantime I have found the outputs something as you suggested
const modelInfo = await tf.node.getMetaGraphsFromSavedModel('./toxicity_model/saved');
console.dir(modelInfo[0].signatureDefs.serving_default.outputs, { depth: null, maxArrayLength: null });
@loretoparisi btw, one advantage of working with saved_model
and getMetaGraphsFromSavedModel()
is that it shows actual signature names instead just an incrementing array (when model has multiple inputs and/or outputs) that you get from a graph_model
.
See https://github.com/tensorflow/tfjs/issues/3942 for details.
@loretoparisi Interesting. I used the following code and it worked just fine:
const tf = require('@tensorflow/tfjs-node') async function run() { const model = await tf.node.loadSavedModel('./models/toxicity_saved/') // both indexArray and valueArray are obtained from two preprocessed test phrases that I used to verify // model outputs const indexArray = [ [0, 1], [0,2 ], [0, 3], [0, 4], [0, 5], [0, 6], [0, 7], [0, 8], [1, 0], [1, 1], [1, 2], [1, 3] ] const valueArray = [215, 13, 53, 4461, 2951, 519, 1129, 7, 78, 16, 123, 20, 6] const indices = tf.tensor2d(indexArray, [indexArray.length, 2], 'int32') const values = tf.tensor1d(valueArray, 'int32') const modelInputs = { Placeholder_1: indices, Placeholder: values } const labels = model.predict(modelInputs) indices.dispose() values.dispose() outputs = [] for (name in labels) { const prediction = labels[name].dataSync() const results = [] for (let input = 0; input < 2; ++input) { const probs = prediction.slice(input * 2, input * 2 + 2) let match = null if (Math.max(probs[0], probs[1]) > 0.9) { match = probs[0] > probs[1] } p= probs.toString() // just to print out the numbers results.push({p, match}) } outputs.push({label: name.split('/')[0], results}) } for (x of outputs) { console.log(x) } } run()
The model methods
outputs()
andinputs()
aren't implemented yet for theSavmedModel
-class, but in case you need them for some reason, outputs and inputs can be obtained using thegetMetaGraphsFromSavedModel()
andgetSignatureDefEntryFromMetaGraphInfo()
functions.
@patlevin thanks for the test script. I modified it a little bit to user the Universal Sentence Encoder:
const use = require("@tensorflow-models/universal-sentence-encoder");
const model = await tf.node.loadSavedModel('./toxicity_model/saved');
const tokenizer = await use.load();
const sentences = ['you suck', 'hello how are you?'];
var codes = await tokenizer.embed(sentences);
console.log(codes);
Now I have an array of Tensors like
Tensor {
kept: false,
isDisposedInternal: false,
shape: [ 1, 512 ],
dtype: 'float32',
size: 512,
strides: [ 512 ],
dataId: {},
id: 837,
rankType: '2',
scopeId: 1289
}
and now I have to turn into indexes and values. I have tried this
var encodings = await tokenizer.embed(sentences);
var indicesArr = encodings.map(function (arr, i) { return arr.map(function (d, index) { return [i, index]; }); });
var flattenedIndicesArr = [];
for (i = 0; i < indicesArr.length; i++) {
flattenedIndicesArr =
flattenedIndicesArr.concat(indicesArr[i]);
}
var indices = tf.tensor2d(flattenedIndicesArr, [flattenedIndicesArr.length, 2], 'int32');
var values = tf.tensor1d(tf.util.flatten(encodings), 'int32');
But it seems that embed
function of the Tokenizers is different from the encode
function internally used that I cannot call external (it gives me an undefined
if I try tokenizer.encode
), so in this case I get an error:
TypeError: encodings.map is not a function
Thank you.
@loretoparisi The model expects an encoded word vector as input, while the Universal Sentence Encoder (USE) model returns embeddings.
Basically, you'll want to use the loadTokenizer()
function from the previous USE version, but that one requires TFJS 1.x...
I have a working version locally, but it'd be better to fix the examples instead - see Issue model repo
Unfortunately @pyu10055's commit b02310745ceac6b8e4a475719c343da53e3cade2 on the USE-repo broke both the Toxicity example model and your use-case entirely...
The real problem is that the examples are outdated and some changes broke TFJS 2.x compatibility (in the case of USE I fail to see the reasoning behind the change - might have been a mistake?).
Meanwhile, I'll create a gist for you that contains all you need to get this working as a single-file solution. I'll get back to you in a bit.
EDIT: I got confused here, since a similar issue was raised w.r.t. outdated tfjs-examples. The same applies to tfjs-models, though - basically some models are incompatible with TFJS 2.x due to package changes (not for technical reasons).
@patlevin thank you! In fact I have just realized that the after recent updates models examples and core code diverged!
I have the same issue, running node v15.7.0, tfjs-node v3.6.1 on Mac Big Sur
const image = fs.readFileSync(imagePath);
const decodedImage = tfnode.node.decodeImage(new Uint8Array(image), 3);
const model = await tfnode.node.loadSavedModel(modelPath);
const predictions = model.predict(decodedImage);
2021-05-31 18:58:58.942711: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: saved_model
2021-05-31 18:58:59.291324: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-05-31 18:58:59.291368: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: saved_model
2021-05-31 18:59:00.692648: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-05-31 18:59:03.160455: I tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: saved_model
2021-05-31 18:59:04.360189: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 5417477 microseconds.
2021-05-31 18:59:12.511652: E tensorflow/core/framework/tensor.cc:555] Could not decode variant with type_name: "tensorflow::TensorList". Perhaps you forgot to register a decoder via REGISTER_UNARY_VARIANT_DECODE_FUNCTION?
2021-05-31 18:59:12.511706: W tensorflow/core/framework/op_kernel.cc:1740] OP_REQUIRES failed at constant_op.cc:79 : Invalid argument: Cannot parse tensor from tensor_proto.
2021-05-31 18:59:12.546008: E tensorflow/core/framework/tensor.cc:555] Could not decode variant with type_name: "tensorflow::TensorList". Perhaps you forgot to register a decoder via REGISTER_UNARY_VARIANT_DECODE_FUNCTION?
2021-05-31 18:59:12.546526: W tensorflow/core/framework/op_kernel.cc:1740] OP_REQUIRES failed at constant_op.cc:79 : Invalid argument: Cannot parse tensor from proto: dtype: DT_VARIANT
tensor_shape {
}
variant_val {
type_name: "tensorflow::TensorList"
metadata: "\001\000\001\377\377\377\377\377\377\377\377\377\001\030\001"
}
/test-saved-model/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:449
var outputMetadata = this.binding.runSavedModel(id, this.getMappedInputTensorIds(inputs, inputTensorInfos), inputTensorInfos.map(function (info) { return info.name; }).join(','), outputOpNames.join(','));
^
Error: Session fail to run with error: Cannot parse tensor from proto: dtype: DT_VARIANT
tensor_shape {
}
variant_val {
type_name: "tensorflow::TensorList"
metadata: "\001\000\001\377\377\377\377\377\377\377\377\377\001\030\001"
}
Simply calling
tfnode.node.getMetaGraphsFromSavedModel(path);
on a model usinguint8
results in error:However, support for
unint8
was added totfjs
via https://github.com/tensorflow/tfjs/pull/2981 back in March.Those newly supported data types should be added throughout
tfjs
codebase.Environment: Ubuntu 20.04 running NodeJS 14.9.0 with TFJS 2.3.0