its-ogawa commented 4 years ago

problem

What is the difference between the three resize methods (cropAndResize, resizeNearestNeighbor and resizeBilinear)? CropAndResize and resizeNearestNeighbor do not return the expected results. Why is the bottom half of my image lost?

background

We would like to develop an application that performs image processing in Tensorflow.js. To do so, we need to match the size of the image to be input to the trained model. The image to be input is large, so I want to simply scale it down to 128x128. I have found 3 resize methods on the following page, but they do not give me the results I expect. The bottom half of the image is lost. Does anyone have any good ideas?

code

This is the source code I wrote.

<!DOCTYPE html>
<html>
<head></head>
<meta charset="utf-8"/>
<body>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs/dist/tf.min.js"></script>
<canvas id="canvas1" width="256" height="256" style="border: 2px solid;"></canvas>
<canvas id='canvas2' width="128" height="128" style="border: 2px solid;"></canvas>
<canvas id='canvas3' width="128" height="128" style="border: 2px solid;"></canvas>
<canvas id='canvas4' width="128" height="128" style="border: 2px solid;"></canvas>
<script>

const canvas1 = document.getElementById('canvas1');
const canvas2 = document.getElementById('canvas2');
const canvas3 = document.getElementById('canvas3');
const canvas4 = document.getElementById('canvas4');

const SIZE = 128;
const COL_CHANNEL = 3;

const inputImg = new Image(); inputImg.src = "../00_resources/woman172.jpg";
const inCtx = canvas1.getContext('2d');

// image1 (original image)
inCtx.drawImage(inputImg, 0, 0, canvas1.width, canvas1.height);
let img1 = tf.browser.fromPixels(canvas1, COL_CHANNEL);

// image2 (resized image - using 'cropAndResize')
let img2 = tf.image.cropAndResize(img1.expandDims(0), [[0, 0, 1, 1]], [0], [SIZE,SIZE]).div(tf.scalar(255)).squeeze();
tf.browser.toPixels(img2, canvas2);// NG

// image3 (resized image - using 'resizeNearestNeighbor')
let img3 = tf.image.resizeNearestNeighbor(img1, [SIZE, SIZE]).toFloat().div(tf.scalar(255));
tf.browser.toPixels(img3, canvas3);// NG

// image4 (resized image - using 'resizeBilinear')
let img4 = tf.image.resizeBilinear(img1, [SIZE, SIZE]).toFloat().div(tf.scalar(255));
tf.browser.toPixels(img4, canvas4);// OK?

</script>
</body>
</html>

Capturing the result

From left to right.

Original Image.
Resized image using cropAndResize (half of the image is black)
Resized image using resizeNearestNeighbor (half of the image is black)
Resized image using resizeBilinear (appears to be the correct result)

reference

https://js.tensorflow.org/api/latest/

tafsiri commented 4 years ago

I wasn't able to reproduce this exactly, but did run into problem if I didn't wait for the image to finish loading. Could you try the following?

const inputImg = new Image(); 

inputImg.onload = function () {
  const inCtx = canvas1.getContext('2d');

  // image1 (original image)
  inCtx.drawImage(inputImg, 0, 0, canvas1.width, canvas1.height);
  let img1 = tf.browser.fromPixels(canvas1, COL_CHANNEL);

  // image2 (resized image - using 'cropAndResize')
  let img2 = tf.image.cropAndResize(img1.expandDims(0), [[0, 0, 1, 1]], [0], [SIZE,SIZE]).div(tf.scalar(255)).squeeze();
  tf.browser.toPixels(img2, canvas2);// NG

  // image3 (resized image - using 'resizeNearestNeighbor')
  let img3 = tf.image.resizeNearestNeighbor(img1, [SIZE, SIZE]).toFloat().div(tf.scalar(255));
  tf.browser.toPixels(img3, canvas3);// NG

  // image4 (resized image - using 'resizeBilinear')
  let img4 = tf.image.resizeBilinear(img1, [SIZE, SIZE]).toFloat().div(tf.scalar(255));
  tf.browser.toPixels(img4, canvas4);// OK?
}

inputImg.src = "your url here" ;

Notice that here we wait for the onload event on the image before trying to draw it or read it. If that doesn't work could you paste a codepen or glitch example that we can use to try and reproduce the issue.

My attempt to reproduce this:

https://glitch.com/edit/#!/humorous-silky-hacksaw?path=index.html%3A12%3A23 (code) https://humorous-silky-hacksaw.glitch.me (output)

its-ogawa commented 4 years ago

Dear tafsiri,

Thank you for your reply.

Your point is correct. The lack of onload in the code in the first post is my mistake. My apologies.

Back on topic, I am wondering why image2 and image3 are not fully drawn. It's reproduced in the example you gave me.

It doesn't draw image2 and image3 despite image1 being drawn and image4 attempted as well is drawn. Why is this? Can you help me?

Thank you and best regards,

tafsiri commented 4 years ago

Could you send me an online example (in codepen, glitch, or something similar) that demonstrates the problem? I am no able to reproduce it. This is what i see in the link I posted above.

Screen Shot 2020-09-30 at 11 15 01 AM

its-ogawa commented 4 years ago

Thank you for your reply.

I was surprised to hear that it can't be reproduced, but I'm fortunate that you attached the image. I seem to be seeing different results with the same source code.

For example, the image looks incomplete to me even in the link you provided to me. I'll send you the image I'm seeing.

Even in the source code, I'm using what you contributed to glitch. (Glitch is very useful. I got an account there too.) That's my first source code with an ’onload' added to it, but The result looks as imperfect to me as what I see in the link you provided.

Is it an environmental issue? Is there anything else I can offer besides the source code? It may be a very difficult problem, but please help me.

Thank you and best regards,

its-ogawa commented 4 years ago

I'll report what I've found.

it doesn't seem to be due to the type of browser
it doesn't seem to be due to the tfjs version

About 1.

I checked with three different browsers - firefox, chrome and edge - and the results were the same. (I'm attaching a picture just in case) All browsers are kept up to date.

About 2.

I specified the version of tensorflowjs explicitly. For example, to specify the latest version 2.4.0, I used the following

What I've found here is that the same phenomenon exists if you go back to 1.1.2. I've attached an image of this as well.

I don't know if it's related, but if you go back to 0.15.3, 'resizeBilinear' is also incomplete in the same way. I've attached an image of this as well.

It might give you a clue if there was a feature update for 'resizeBilinear' in the 0.15.3 to 1.1.2 versions.

google-ml-butler[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 dyas if no further activity occurs. Thank you.

its-ogawa commented 4 years ago

@tafsiri @rthadur

This post has not yet been resolved. Please do not close it. 'Installed' is not an appropriate action.

Also, I have presented the information I can report. 'stat:awiting response' is also not appropriate.

Please reply if there is anything I can do to help.

tafsiri commented 4 years ago

@rthadur are you able to reproduce the issue described here when visiting this link https://humorous-silky-hacksaw.glitch.me/

rthadur commented 4 years ago

Could you send me an online example (in codepen, glitch, or something similar) that demonstrates the problem? I am no able to reproduce it. This is what i see in the link I posted above.

@tafsiri I am seeing the same in all browsers.

tafsiri commented 3 years ago

Hi @its-ogawa, from looking at this and a few other issues we are beginning to suspect an issue with particular integrated graphics chipsets, could you tell us what hardware/graphics card you are running on?

Could you also post a screenshot of what you see here https://js.tensorflow.org/debug/

its-ogawa commented 3 years ago

@tafsiri Dear tafsiri,

Thank you for your reply. I'm glad you are continuing your research.

I'll send you the hardware/graphics card information. Please check the attached "CPU-Z_report.txt". CPU-Z_report.txt

Also attached is a capture of the results of the Debug info (https://js.tensorflow.org/debug/).

I hope this helps with the solution. I know it's a lot of work, but please help me out.

Thank you and best regards,

tafsiri commented 3 years ago

Thanks, just pulling out the graphics card info for other to see.

Display adapter 0
ID 0x4000000 Name Intel(R) HD Graphics 4000 Board Manufacturer 0x10CF (0x176D) PCI device bus 0 (0x0), device 2 (0x2), function 0 (0x0) Vendor ID 0x8086 (0x10CF) Model ID 0x0162 (0x176D) Revision ID 0x9 Performance Level 0

cc @annxingyuan @pyu10055 for comparison with other related issues.

rthadur commented 2 years ago

can you please try with latest version , if the issue still exists ?

its-ogawa commented 2 years ago

@rthadur Thank you for contacting us. I am glad that you continue to work on this issue.

Now, as for the results, resizeNearestNeighbor works, but cropAndResize does not.

The source code is the same as the old one.

I think tfjs has been uploaded. I am running the following command.

>npm install -g tfjs
+ tfjs@0.6.0
added 4 packages from 4 contributors in 4.044s

>npm install -g @tensorflow/tfjs
C:\Users\ogawa\AppData\Roaming\npm\tfjs-custom-module -> C:\Users\ogawa\AppData\Roaming\npm\node_modules\@tensorflow\tfjs\dist\tools\custom_module\cli.js

> core-js@3.19.1 postinstall C:\Users\ogawa\AppData\Roaming\npm\node_modules\@tensorflow\tfjs\node_modules\core-js
> node -e "try{require('./postinstall')}catch(e){}"

Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!

The project needs your help! Please consider supporting of core-js:
> https://opencollective.com/core-js
> https://patreon.com/zloirock
> https://paypal.me/zloirock
> bitcoin: bc1qlea7544qtsmj2rayg0lthvza9fau63ux0fstcz

Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)

+ @tensorflow/tfjs@3.11.0
added 49 packages from 84 contributors in 16.554s

The result of this link is as follows.

google-ml-butler[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

its-ogawa commented 2 years ago

@tafsiri @rthadur

This post has not yet been resolved. Please do not close it. I have presented the information I can report.

Please reply if there is anything I can do to help.

gaikwadrahul8 commented 1 year ago

Hi, @its-ogawa

Apologize for the delayed response and It seems like this issue has been taken care with latest version of @tensorflow/tfjs@4.4.0 and I also tried with @tensorflow/tfjs@4.2.0 It working as expected if I'm not wrong and I have added screenshot below for your reference so I would request you to please try with latest version of @tensorflow/tfjs and check whether your issue is resolving or not ?

If issue still persists please let us know ? Thank you!

its-ogawa commented 1 year ago

@gaikwadrahul8 Thanks for reply. I am very happy to hear the news! I tried the same with Glitch, but 'cropAndResize' doesn't seem to be working. Why is that?

gaikwadrahul8 commented 1 year ago

Hi, @its-ogawa

Apologize for the delayed response and I tried to replicate the same issue on my Mac M1 but I haven't observed same issue and I have attached screenshot above so I'm not sure at the moment whether this issue is related with specific hardware or not so May I know on which system did try same Glitch where cropAndResize doesn't seem to be working please ? Thank you!

CC :@jinjingforever

its-ogawa commented 1 year ago

Hi, @gaikwadrahul8

Thanks for getting back to me. My operating system is Windows. By the way, my browser is Google Chrome. I hope this is helpful.

Windows 10 Pro version 22H2 
Google Chrome version 113.0.5672.93

Pensarfeo commented 10 months ago

@its-ogawa

Working on Linux, had no issue. I was wondering if your problem is not a tfjs one but a browser one? It has happened to me that images where only half printed to canvas.

Have you tried to manually take the data from the crop output and only print the bottom half, or even console.log the output to console to check if those numbers are all 0?

its-ogawa commented 10 months ago

@Pensarfeo

Dear Pensarfeo Thank you for contacting.

Is it a browser issue? As I commented below, it reproduces in my environment regardless of the browser. https://github.com/tensorflow/tfjs/issues/3945#issuecomment-701883931

I tried outputting the crop output to the console. The values seem to be 0. I am attaching a capture. console

Can you tell me what environment you checked? Also, is the code you ran the same as the one shown below? https://github.com/tensorflow/tfjs/issues/3945#issue-704196383

https://github.com/tensorflow/tfjs/issues/3945#issuecomment-700779879

its-ogawa commented 10 months ago

I will post the entire source code again, just to be sure.

<!DOCTYPE html>
<html>
<head></head>
<meta charset="utf-8"/>
<body>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.2.0/dist/tf.min.js"></script>
<canvas id="canvas1" width="256" height="256" style="border: 2px solid;"></canvas>
<canvas id='canvas2' width="128" height="128" style="border: 2px solid;"></canvas>
<canvas id='canvas3' width="128" height="128" style="border: 2px solid;"></canvas>
<canvas id='canvas4' width="128" height="128" style="border: 2px solid;"></canvas>
<script>

const canvas1 = document.getElementById('canvas1');
const canvas2 = document.getElementById('canvas2');
const canvas3 = document.getElementById('canvas3');
const canvas4 = document.getElementById('canvas4');

const SIZE = 128;
const COL_CHANNEL = 3;

const inputImg = new Image(); 

// image1 (original image)
inputImg.crossOrigin = "anonymous"; // for CORS
const inCtx = canvas1.getContext('2d');
inputImg.src = "https://cdn.glitch.com/5f39d744-9532-4d43-bc1c-6d20e987b956%2Fcat.d8f4b943.jpg?v=1563376047812" ;
inputImg.onload = function() {
    inCtx.drawImage(inputImg, 0, 0, canvas1.width, canvas1.height);

    let img1 = tf.browser.fromPixels(canvas1, COL_CHANNEL);
    console.log('img1')
    console.log(img1.print());

    // image2 (resized image - using 'cropAndResize')
    let img2 = tf.image.cropAndResize(img1.expandDims(0), [[0, 0, 1, 1]], [0], [SIZE,SIZE]).div(tf.scalar(255)).squeeze();
    tf.browser.toPixels(img2, canvas2);// NG

    console.log('img2')
    console.log(img2.print());

    // image3 (resized image - using 'resizeNearestNeighbor')
    let img3 = tf.image.resizeNearestNeighbor(img1, [SIZE, SIZE]).toFloat().div(tf.scalar(255));
    tf.browser.toPixels(img3, canvas3);// NG

    console.log('img3')
    console.log(img3.print());

    // image4 (resized image - using 'resizeBilinear')
    let img4 = tf.image.resizeBilinear(img1, [SIZE, SIZE]).toFloat().div(tf.scalar(255));
    tf.browser.toPixels(img4, canvas4);// OK?

    console.log('img4')
    console.log(img4.print());
};

</script>
</body>
</html>

Pensarfeo commented 9 months ago

@its-ogawa Tried on chrome on linux.

gaikwadrahul8 commented 9 months ago

Hi, @its-ogawa

I apologize for the delayed response and I tried above shared code-snippet on my Windows 10 (VM) and it seems like working as expected for your reference I have added screenshot below so could you please try with below mentioned Google chrome version and see is it working as expected or not ?

If issue still persists please let us know to investigate this issue further. Thank you for your understanding and patience.

Google Chrome Version 120.0.6099.130 (Official Build) (64-bit)

its-ogawa commented 9 months ago

@gaikwadrahul8 Thanks for your reply.

I tried the following version of Chrome, unfortunately with the same result. I updated Chrome to the latest version to try it, but still the same result.

What could be the reason why cropAndResize is not enabled? Please help.

(And a Happy New Year to you too. I wish you a great year :)

tensorflow / tfjs

The behavior of the three resize methods (cropAndResize, resizeNearestNeighbor and resizeBilinear) #3945

problem

background

code

reference