ValueError: Found more than one (2) load handlers

aprudnikoff commented 1 month ago

Describe the bug I'm already using upscalerJS/node locally on macbook@cpu and it works fine. Now I would like to improve performance and run my setup on vast.io server at GPU but got stuck with an issue. This is my code simplified as much as possible. Indeed it's pretty equal to an example from the website.

const fs = require('fs');
const model = require('@upscalerjs/esrgan-thick/3x');
const tf = require('@tensorflow/tfjs-node-gpu');
const Upscaler = require('upscaler/node');

async function run() {
  const sourceFileName = './images/source/src-face.jpg';
  const targetFileName = './images/source/upscaled.jpg';
  const image = tf.node.decodeJpeg(fs.readFileSync(sourceFileName), 3);

  const upscaler = new Upscaler({ model });
  const tensor = await upscaler.upscale(image);

  const upscaledTensorJpg = await tf.node.encodeJpeg(tensor, '', 100);
  fs.writeFileSync(targetFileName, upscaledTensorJpg);
}

run();

But I get the following exceptional output:

root@C.13160681:~/projects/stock-ass$ node ./research/upscale.js
2024-10-15 20:28:08.478402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7386 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1070 Ti, pci bus id: 0000:82:00.0, compute capability: 6.1
The kernel 'FFT' for backend 'tensorflow' is already registered
The kernel 'IFFT' for backend 'tensorflow' is already registered
The kernel 'LRN' for backend 'tensorflow' is already registered
The kernel 'LRNGrad' for backend 'tensorflow' is already registered
The kernel '_FusedMatMul' for backend 'tensorflow' is already registered
The kernel 'Abs' for backend 'tensorflow' is already registered
The kernel 'Acos' for backend 'tensorflow' is already registered
The kernel 'Acosh' for backend 'tensorflow' is already registered
The kernel 'Add' for backend 'tensorflow' is already registered
The kernel 'AddN' for backend 'tensorflow' is already registered
The kernel 'All' for backend 'tensorflow' is already registered
The kernel 'Any' for backend 'tensorflow' is already registered
The kernel 'ArgMax' for backend 'tensorflow' is already registered
The kernel 'ArgMin' for backend 'tensorflow' is already registered
The kernel 'Asin' for backend 'tensorflow' is already registered
The kernel 'Asinh' for backend 'tensorflow' is already registered
The kernel 'Atan2' for backend 'tensorflow' is already registered
The kernel 'Atan' for backend 'tensorflow' is already registered
The kernel 'Atanh' for backend 'tensorflow' is already registered
The kernel 'AvgPool3D' for backend 'tensorflow' is already registered
The kernel 'AvgPool3DGrad' for backend 'tensorflow' is already registered
The kernel 'AvgPool' for backend 'tensorflow' is already registered
The kernel 'AvgPoolGrad' for backend 'tensorflow' is already registered
The kernel 'BatchMatMul' for backend 'tensorflow' is already registered
The kernel 'BatchToSpaceND' for backend 'tensorflow' is already registered
The kernel 'BroadcastArgs' for backend 'tensorflow' is already registered
The kernel 'Cast' for backend 'tensorflow' is already registered
The kernel 'Ceil' for backend 'tensorflow' is already registered
The kernel 'ClipByValue' for backend 'tensorflow' is already registered
The kernel 'ComplexAbs' for backend 'tensorflow' is already registered
The kernel 'Complex' for backend 'tensorflow' is already registered
The kernel 'Concat' for backend 'tensorflow' is already registered
The kernel 'Conv2DBackpropFilter' for backend 'tensorflow' is already registered
The kernel 'Conv2DBackpropInput' for backend 'tensorflow' is already registered
The kernel 'Conv2D' for backend 'tensorflow' is already registered
The kernel 'Conv3DBackpropFilterV2' for backend 'tensorflow' is already registered
The kernel 'Conv3DBackpropInputV2' for backend 'tensorflow' is already registered
The kernel 'Conv3D' for backend 'tensorflow' is already registered
The kernel 'Cos' for backend 'tensorflow' is already registered
The kernel 'Cosh' for backend 'tensorflow' is already registered
The kernel 'CropAndResize' for backend 'tensorflow' is already registered
The kernel 'Cumprod' for backend 'tensorflow' is already registered
The kernel 'Cumsum' for backend 'tensorflow' is already registered
The kernel 'Bincount' for backend 'tensorflow' is already registered
The kernel 'DepthToSpace' for backend 'tensorflow' is already registered
The kernel 'DepthwiseConv2dNativeBackpropFilter' for backend 'tensorflow' is already registered
The kernel 'DepthwiseConv2dNativeBackpropInput' for backend 'tensorflow' is already registered
The kernel 'DepthwiseConv2dNative' for backend 'tensorflow' is already registered
The kernel 'Diag' for backend 'tensorflow' is already registered
The kernel 'Dilation2DBackpropFilter' for backend 'tensorflow' is already registered
The kernel 'Dilation2DBackpropInput' for backend 'tensorflow' is already registered
The kernel 'Dilation2D' for backend 'tensorflow' is already registered
The kernel 'Elu' for backend 'tensorflow' is already registered
The kernel 'EluGrad' for backend 'tensorflow' is already registered
The kernel 'Einsum' for backend 'tensorflow' is already registered
The kernel 'Equal' for backend 'tensorflow' is already registered
The kernel 'Erf' for backend 'tensorflow' is already registered
The kernel 'Exp' for backend 'tensorflow' is already registered
The kernel 'ExpandDims' for backend 'tensorflow' is already registered
The kernel 'Expm1' for backend 'tensorflow' is already registered
The kernel 'Fill' for backend 'tensorflow' is already registered
The kernel 'FlipLeftRight' for backend 'tensorflow' is already registered
The kernel 'Floor' for backend 'tensorflow' is already registered
The kernel 'FloorDiv' for backend 'tensorflow' is already registered
The kernel 'FusedBatchNorm' for backend 'tensorflow' is already registered
The kernel 'FusedConv2D' for backend 'tensorflow' is already registered
The kernel 'FusedDepthwiseConv2D' for backend 'tensorflow' is already registered
The kernel 'GatherNd' for backend 'tensorflow' is already registered
The kernel 'GatherV2' for backend 'tensorflow' is already registered
The kernel 'Greater' for backend 'tensorflow' is already registered
The kernel 'GreaterEqual' for backend 'tensorflow' is already registered
The kernel 'Identity' for backend 'tensorflow' is already registered
The kernel 'Imag' for backend 'tensorflow' is already registered
The kernel 'IsFinite' for backend 'tensorflow' is already registered
The kernel 'IsInf' for backend 'tensorflow' is already registered
The kernel 'IsNan' for backend 'tensorflow' is already registered
The kernel 'LeakyRelu' for backend 'tensorflow' is already registered
The kernel 'Less' for backend 'tensorflow' is already registered
The kernel 'LessEqual' for backend 'tensorflow' is already registered
The kernel 'LinSpace' for backend 'tensorflow' is already registered
The kernel 'Log1p' for backend 'tensorflow' is already registered
The kernel 'Log' for backend 'tensorflow' is already registered
The kernel 'LogicalAnd' for backend 'tensorflow' is already registered
The kernel 'LogicalNot' for backend 'tensorflow' is already registered
The kernel 'LogicalOr' for backend 'tensorflow' is already registered
The kernel 'Max' for backend 'tensorflow' is already registered
The kernel 'MaxPool3D' for backend 'tensorflow' is already registered
The kernel 'MaxPool3DGrad' for backend 'tensorflow' is already registered
The kernel 'MaxPool' for backend 'tensorflow' is already registered
The kernel 'MaxPoolGrad' for backend 'tensorflow' is already registered
The kernel 'Maximum' for backend 'tensorflow' is already registered
The kernel 'Mean' for backend 'tensorflow' is already registered
The kernel 'Min' for backend 'tensorflow' is already registered
The kernel 'Minimum' for backend 'tensorflow' is already registered
The kernel 'MirrorPad' for backend 'tensorflow' is already registered
The kernel 'Mod' for backend 'tensorflow' is already registered
The kernel 'Multinomial' for backend 'tensorflow' is already registered
The kernel 'Multiply' for backend 'tensorflow' is already registered
The kernel 'Neg' for backend 'tensorflow' is already registered
The kernel 'NonMaxSuppressionV3' for backend 'tensorflow' is already registered
The kernel 'NonMaxSuppressionV4' for backend 'tensorflow' is already registered
The kernel 'NonMaxSuppressionV5' for backend 'tensorflow' is already registered
The kernel 'NotEqual' for backend 'tensorflow' is already registered
The kernel 'OneHot' for backend 'tensorflow' is already registered
The kernel 'OnesLike' for backend 'tensorflow' is already registered
The kernel 'Pack' for backend 'tensorflow' is already registered
The kernel 'PadV2' for backend 'tensorflow' is already registered
The kernel 'Pow' for backend 'tensorflow' is already registered
The kernel 'Prelu' for backend 'tensorflow' is already registered
The kernel 'Prod' for backend 'tensorflow' is already registered
The kernel 'Range' for backend 'tensorflow' is already registered
The kernel 'Real' for backend 'tensorflow' is already registered
The kernel 'RealDiv' for backend 'tensorflow' is already registered
The kernel 'Reciprocal' for backend 'tensorflow' is already registered
The kernel 'Relu6' for backend 'tensorflow' is already registered
The kernel 'Relu' for backend 'tensorflow' is already registered
The kernel 'Reshape' for backend 'tensorflow' is already registered
The kernel 'ResizeBilinear' for backend 'tensorflow' is already registered
The kernel 'ResizeBilinearGrad' for backend 'tensorflow' is already registered
The kernel 'ResizeNearestNeighbor' for backend 'tensorflow' is already registered
The kernel 'ResizeNearestNeighborGrad' for backend 'tensorflow' is already registered
The kernel 'Reverse' for backend 'tensorflow' is already registered
The kernel 'Round' for backend 'tensorflow' is already registered
The kernel 'Rsqrt' for backend 'tensorflow' is already registered
The kernel 'ScatterNd' for backend 'tensorflow' is already registered
The kernel 'Select' for backend 'tensorflow' is already registered
The kernel 'Selu' for backend 'tensorflow' is already registered
The kernel 'Sigmoid' for backend 'tensorflow' is already registered
The kernel 'Sign' for backend 'tensorflow' is already registered
The kernel 'Sin' for backend 'tensorflow' is already registered
The kernel 'Sinh' for backend 'tensorflow' is already registered
The kernel 'Slice' for backend 'tensorflow' is already registered
The kernel 'Softmax' for backend 'tensorflow' is already registered
The kernel 'Softplus' for backend 'tensorflow' is already registered
The kernel 'SpaceToBatchND' for backend 'tensorflow' is already registered
The kernel 'SparseToDense' for backend 'tensorflow' is already registered
The kernel 'SplitV' for backend 'tensorflow' is already registered
The kernel 'Sqrt' for backend 'tensorflow' is already registered
The kernel 'Square' for backend 'tensorflow' is already registered
The kernel 'SquaredDifference' for backend 'tensorflow' is already registered
The kernel 'StaticRegexReplace' for backend 'tensorflow' is already registered
The kernel 'Step' for backend 'tensorflow' is already registered
The kernel 'StridedSlice' for backend 'tensorflow' is already registered
The kernel 'Sub' for backend 'tensorflow' is already registered
The kernel 'Sum' for backend 'tensorflow' is already registered
The kernel 'Tan' for backend 'tensorflow' is already registered
The kernel 'Tanh' for backend 'tensorflow' is already registered
The kernel 'TensorScatterUpdate' for backend 'tensorflow' is already registered
The kernel 'Tile' for backend 'tensorflow' is already registered
The kernel 'TopK' for backend 'tensorflow' is already registered
The kernel 'Transpose' for backend 'tensorflow' is already registered
The kernel 'Unique' for backend 'tensorflow' is already registered
The kernel 'Unpack' for backend 'tensorflow' is already registered
The kernel 'UnsortedSegmentSum' for backend 'tensorflow' is already registered
The kernel 'ZerosLike' for backend 'tensorflow' is already registered
2024-10-15 20:28:08.613607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7386 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1070 Ti, pci bus id: 0000:82:00.0, compute capability: 6.1
tensorflow backend was already registered. Reusing existing backend factory.
/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:273
        var _this = _super.call(this, message) || this;
                           ^

ValueError: Found more than one (2) load handlers for URL 'file:///root/projects/stock-ass/node_modules/@upscalerjs/esrgan-thick/models/x2/model.json'
    at new ValueError (/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:273:28)
    at Object.<anonymous> (/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:25743:27)
    at step (/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:159:27)
    at Object.next (/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:108:53)
    at /root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:101:71
    at new Promise (<anonymous>)
    at __awaiter (/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:87:12)
    at Object.loadLayersModel (/root/projects/stock-ass/node_modules/@tensorflow/tfjs-layers/dist/tf-layers.node.js:25727:12)
    at loadTfModel (/root/projects/stock-ass/node_modules/upscaler/dist/node/upscalerjs/src/shared/model-utils.js:89:15)
    at /root/projects/stock-ass/node_modules/upscaler/dist/node/upscalerjs/src/node/loadModel.node.js:92:58

Node.js v20.18.0

Expected behavior Upscaling should work as expected without errors.

Additional details On vast.ai I use the following image as a base nvidia/cuda:11.6.1-runtime-ubuntu20.04 And then add some things that are missing in the base image:

# CUDA
export LD_LIBRARY_PATH=/usr/local/cuda-11.6/targets/x86_64-linux/lib

# Install CUDNN
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
export last_public_key=3bf863cc
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/${last_public_key}.pub
sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
sudo apt-get update
sudo apt-get install libcudnn8=8.4.1.50-1+cuda11.6
sudo apt-get install libcudnn8-dev=8.4.1.50-1+cuda11.6

# Install NVM
# Install node
# Clone my repo
# npm install
# npm start

thekevinscott commented 1 month ago

The lines about x for backend 'tensorflow' is already registered is likely because you're importing both a -node-gpu and -node version:

const tf = require('@tensorflow/tfjs-node-gpu');
const Upscaler = require('upscaler/node');

Update Upscaler to be the node-gpu flavor:

const tf = require('@tensorflow/tfjs-node-gpu');
const Upscaler = require('upscaler/node-gpu');

However, that probably won't resolve the second part of your error, which I've never seen before.

I tried reproducing the error on Vast AI, however, I'm getting a different error: TypeError: (0 , util_1.isNullOrUndefined) is not a function. Is there a way that you could turn your Dockerfile into an image that you could share, so that we're definitely working off the same system?

Other ideas:

Does this error happen with 2x or 4x models?
If you omit the model (i.e., use the default model), does the same error appear? What about other models like esrgan-slim or esrgan-medium?
Try doing a console.log(model) to see what it reports.

aprudnikoff commented 2 weeks ago

The only one simple change made the trick for me, require('upscaler/node-gpu') As I remember, it wasn't clearly documented. Can't check now since website seems down. Thank you very much for clarification.

thekevinscott / UpscalerJS

ValueError: Found more than one (2) load handlers #1296