rthadur
commented
3 years ago The above method should work , I believe this is a bug
Open HDegroote opened 3 years ago
rthadur
commented
3 years ago The above method should work , I believe this is a bug
SahithiMeda
commented
3 years ago I have the same issue with the model.save() method which is giving ReferenceError: FormData is not defined. In the node modules (node_modules\@tensorflow\tfjs-core\dist\tf-core.node.js) form-data is required.
HDegroote
commented
3 years ago If anyone has an intuition as to what the underlying problem could be and could point me in the right direction, I'd be more than happy to try my hand at fixing this
SahithiMeda
commented
3 years ago https://www.npmjs.com/package/form-data says from version 3.x FormData has dropped support for node@4.x. Not sure, but this might be the reason for the reference error.
HDegroote
commented
3 years ago I did some more research, and I think I've got the problem resolved, albeit in an ad-hoc manner. If one of the devs confirms this makes sense, I'll be happy to make a pull request so you can check if it breaks anything else.
The issue is that Node.js does not natively have FormData, while browserside Javascript does. The same applies for Blob.
Two changes are required to fix the issue, both to tf-core.node.js:
const FormData = require('form-data');(note: this might break browerside Javascript code, in case this code is supposed to run browserside as well)
// Use Buffer on Node.js instead of Blob/atob/btoa
var useNodeBuffer = typeof Buffer !== 'undefined' &&
(typeof Blob === 'undefined' || typeof atob === 'undefined' ||
typeof btoa === 'undefined');After these lines I added the following, to explicitly replace all usage of Blob by Buffer:
if(useNodeBuffer){
Blob = Buffer
}Note: at first sight the variable useNodeBuffer was introduced to solve the issue of Blob not being present in Node. However, it seems to be implemented in a way which is prone to issues. I think the original idea was to include an if-statement whenever Blob or something similar was used, and to implement different logic for Node than for browserside Javascript. But obviously it's easy to forget this, which I think is what happened when implementing the logic for creating that http request.
Sources: https://stackoverflow.com/questions/63576988/how-to-use-formdata-in-node-js-without-browser https://stackoverflow.com/questions/14653349/node-js-cant-create-blobs
rthadur
commented
3 years ago @HDegroote thank you for details , will you be interested in submitting a PR ?
HDegroote
commented
3 years ago @rthadur I hadn't realised the original source was in typescript: my debugging was on the generated Javascript file, and my proposed solution for Blob doesn't seem to work with typescript due to different types.
I've never worked with typescript, and I'm struggling because whatever I do I keep getting type errors. Could anyone who knows typescript have a quick look? I think it's a fairly easy fix to make (although probably indicative of a larger issue in the project, because this will happen wherever Blob or FormData is used on Node).
The relevant file is "tfjs-core/src/io/http.ts". The changes to make are:
const FormData = require('form-data'); at the top (and ensure this doesn't break browser-based javascript)async save(modelArtifacts: ModelArtifacts): Promise<SaveResult>=> For the monkeypatch solution see https://stackoverflow.com/questions/14653349/node-js-cant-create-blobs, or simply replace by Buffer as I did in my previous comment => For the explicit 'if', the code should be something like the following (with useNodeBuffer defined as in my previous comment):
if(useNodeBuffer){
init.body.append('model.json', new Buffer.from([JSON.stringify(modelTopologyAndWeightManifest)], { type: JSON_TYPE }), 'model.json');
}
else{
init.body.append('model.json', new Blob([JSON.stringify(modelTopologyAndWeightManifest)], { type: JSON_TYPE }), 'model.json');
}A cleaner solution would probably use a makeBlob function, and put the 'if' there, to reduce code duplication.
In case it's relevant, I'm using the following code to create a server to handle the save_model requests: https://gist.github.com/HDegroote/44bf3bfe919b0a473f4a52ca4ea5b4d6
gaikwadrahul8
commented
1 year ago Hi, @HDegroote
Apologize for the delayed response and I was able to reproduce the issue with version of @tensorflow/tfjs@4.1.0 and it's giving different error message and for your reference I have added screenshot below
TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11413:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
cause: Error: connect ECONNREFUSED 127.0.0.1:3000
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
errno: -61,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 3000
}
}
Node.js v18.15.0Hi, @lina128 Do you have any pointers for this issue please ?
System information
Describe the current behavior When trying to call an http POST endpoint to save a model, a ReferenceError is thrown at @tensorflow/tfjs-core/dist/tf-core.node.js:8270: ReferenceError: FormData is not defined (see log at end of post for full stacktrace)
Describe the expected behavior The code should execute without throwing a ReferenceError
Standalone code to reproduce the issue
Note: I did not use the standard saving code from the docs, which would be
await model.save('http://localhost:3000/save_model')because then I got errors concerning not being able to find save handlers for the URL, although the endpoints were working. I figured this was related to https://github.com/tensorflow/tfjs/issues/723, but could not get it to work with any combination of imports, so opted for this approach. If the explicit usage of tf.io.http makes no sense, feel free to let me know and I'll further debug my initial approach with the standard model.save(url) instead.Note: this is the maximally reduced version of the actual code. The actual code creates an express server with get_model and save_model endpoints. Obviously this minimal example is not working code as the endpoint does not exist, but it already generated the error. I can provide working stand-alone code (up to this bug) with the express server if relevant.
Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Full traceback:
I made a first attempt at debugging by adding the FormData dependency to tf-core.node.js, but then another similar error occurred a few lines later. I think the code is not executing some require statements, but I don't know where it's going wrong.