Are there requirements for the input image size?(question)

[jasan-s]

For the mobile net feature extractor and for using the knn classifier are there any input image requirements? i.e can provide any aspect ratio and resolution image to the mobile net and the knn classifier? Or is it required that a 1:1 ratio image is served but no requirement for the resolution size. Or is there a limit to the max resolution as well.

[joeyklee]

@jasan-s - Great question! Something we should also point to in our documentation.

In general, here's what I would say (@yining1023 feel free to correct me if I'm wrong!):

• resolution @ 72 dpi I think is good/fine. Higher resolution probably won't help you more except make the processing times longer.
• Image size size wise, probably better to not have gigantic images (> 640px x 480px), so smaller the images the more manageable (< 640px x 480px) .

I hope this helps! I will keep this open so that we make sure to add:

1. A note about image resolution and size
2. A note about the number of training data limits (seems also to be a common question that pops up), see: https://github.com/ml5js/ml5-library/issues/405, and https://github.com/ml5js/ml5-library/issues/356

Thank you!

[yining1023]

Hi @jasan-s, Good question! I agree with @joeyklee. There is no specific requirement for image size or resolution.

From the library source code point of view:

• For the mobile net feature extractor: ml5.featureExtractor(‘mobileNet’).addImage(img) will crop the image from the center in 224x224 px.
• For KNN classifier, it doesn't transform the image in any way.

[jasan-s]

@yining1023 @joeyklee agree that it would be great addition to the docs. Since mobile net crops a square 224 px from the center it would be best to enforce the input(i.e live webcam feed) to be a square ratio. Because if the important area of the feature for classification lies outside the square then user adding those examples to model would be meaningless. Would you both agree with the following steps of implementation: 1) Force users webcam feed to be 1:1 square ratio (i.e 512x512) higher res for better user experience 2) user clicks a button to train a class for KNN classifier. The 512x512 image is scaled to 214x214 sent to mobile net feature extractor. And the features are added to knn model. (User still seeing the higher res feed) 3)For testing predictions the user still sees the 512x512 feed and the same scaling down happens.

Important difference is that what user sees on screen is higher resolution but the same 1:1 square ratio then what is being used for building the model.

[shiffman]

Because we want ml5 to be easy for beginners, we made the decision to handle all of the image cropping and resizing behind the scenes when sending an image to MobileNet. However, we probably should be more clear in the documentation as to what is going on as well as consider consistency with other systems. This came up when working with @irealva on compatibility with the new teachable machine which by default flips the webcam's image!

(@yining1023 re: KNN, I believe anytime an image is sent to MobileNet it has to be a 224x224 square image? KNNclassifier can, of course, accept other forms of data, but in terms of the features of an image extracted from MobileNet, the same constraints apply, yes?)

[yining1023]

@shiffman yes! That is correct.

[jasan-s]

@shiffman agree that handling image normalizing within lib is good decision. seems to be done here mobilenetInfer function using image.resize_bilinear.

@shiffman @yining1023 @joeyklee Can you please confirm if the the below statements are true:

Statement: No mater what size and aspect ratio image is inputed to ml5, there is NEVER any image data loss in normalizing and resizing because align_corners is set true?

Result: If above is true then ml5 users don't have to do any image manipulation and can carelessly send any image.

[sproutleaf]

Transferred issue from ml5-library to ml5-website because the question has been answered and now we need to update the documentation.

[devihall]

@sproutleaf > Transferred issue from ml5-library to ml5-website because the question has been answered and now we need to update the documentation.

Not seeing the answers/response to questions from @jasan-s in https://github.com/ml5js/ml5-website/issues/193#issuecomment-1629994362

"Can you please confirm if the the below statements are true:

Statement: No mater what size and aspect ratio image is inputed to ml5, there is NEVER any image data loss in normalizing and resizing because align_corners is set true?

Result: If above is true then ml5 users don't have to do any image manipulation and can carelessly send any image."

Am I missing something? I'm also interested in figuring out the optimal size and resolution for our project