new classification & regression examples

[genekogan]

hey all!

i made some new image classification and regression (transfer learned from mobilenet) demos for a workshop this week, which use tf.js to control p5. see https://ml4a.github.io/demos/tfjs

basically i stripped off pacman from their pacman example for the 2 classification demos (simple + sound), and then modified it to do regression of a single slider (simple + pong). also wrapped the posenet example and having it control p5.sound. ideally ml5 would have classes for custom image classifier & regression at some point (knn is somewhat suboptimal).

would these be useful to have? i may potentially be able to write these into proper classes at some point. my javascript is quite old-school/limited and things like promises/await/=>/yarn are new to me, so may need someone good at javascript to look that over/correct, but i'm happy to sort out some of the junky tensorflow stuff to get some more models available.

let me know what you think of those examples!

[shiffman]

Wow, I just quickly checked these out, awesome! We have the transfer learning stuff in progress for image classification but I hadn't thought about applying it with regression in the way that you have. These would definitely be useful! I'm happy to collaborate with you to turn these into JavaScript classes that work with ml5 if you want to keep plugging away at the tf.js stuff.

[genekogan]

yah i think a simple regression example will definitely be nice to have. yeah i'd be happy to contribute it. where is the in-progress class?

[shiffman]

It just got merged #117. See: https://github.com/ml5js/ml5-library/tree/master/src/ImageClassifier and also #121.

[genekogan]

ah perfect!

a regression class would look almost identical to this. just glancing quickly, i believe the only differences would be:

• kernelInitializer: 'Zeros', activation: 'linear' in the last dense layer
• when you add an example, const y = tf.tidy(() => tf.tensor2d([label])); instead of one-hot.
• optimizer would use loss: 'meanSquaredError' instead of cross-entropy.
• in prediction, return predictions.as1D(); instead of argMax. and no getTopKClasses.

[shiffman]

Cool, do you have any thoughts on the naming? An ImageClassifier makes perfect sense but an ImageRegressor sounds weird and the term "regression" for all its simplicity is intimidating.

• ImageSlider
• ImageRegressor
• ImageRegression

???

Or is it still the ImageClassifier but a flag to turn on regression?

let slider = new ml5.ImageClassifier('MobileNet', 'regression');

[genekogan]

ah yeah tough call. regressor is correct, but i've always found it awkward too. i don't have a strong opinion either way, though i think using it as a flag for ImageClassifier is a bit confusing since it's not classification. what if instead of ImageClassifier you have ImagePredictor with two modes? i.e.

let slider = new ml5.ImagePredictor('MobileNet', 'regression');
let category = new ml5.ImagePredictor('MobileNet', 'classification');

[cvalenzuela]

This look amazing! Regardless of how a new class integration might look, we should also work on adding those examples to the examples repo: https://github.com/ml5js/ml5-examples

In terms of naming, we could also use something like this:

// Create the classifier as usual
let classifier = new ml5.ImageClassifier('Mobilenet');

// So we can still call a classification like this:
classifier.predict(img, callback);

// But then we can also add something that might look like this:
let slider = classifier.asRegression();

// and then add images:
slider.addImage(img, value);

// and then:
slider.predict(img, callback);

[shiffman]

@cvalenzuela I like this asRegression() idea! It makes a lot of sense given that we are starting with the ImageClassifier in the first place. Thanks for this idea @genekogan!

[genekogan]

i'd avoid conflating the two, personally, as regression is not a form of classification or vice-versa, but i don't have a strong opinion on it.

[cvalenzuela]

Agree! Maybe something like

let slider = classifier.withRegression();

That may suggest the idea that the slider uses a classifier together with a regression?

[shiffman]

@genekogan yes, I agree about mixing up the two and being careful about it being confusing!

This is how I'm thinking about it. We start with MobileNet which is a model trained for classification. So the user first creates an ImageClassifier object with MobileNet.

let classifier = ml5.ImageClassifier('MobileNet');

The classifier can be used as is or can be modified with transfer learning (still a classifier). However, another scenario is that a new kind of object -- "regressor" -- can be created from the classifier for transfer learning regression:

let regressor = classifier.asRegression();

The classifier would still just give labels back if predict() is called but now the regressor is can be trained with images and continuous values.

If we were doing a regression right from the beginning I wouldn't want to say "classifier" but these are actually the steps so I think it makes sene? Maybe asRegression() isn't the best terminology.

Some other ideas are:

let classifier = ml5.ImageClassifier('MobileNet');
let regressor = classifier.convertRegression();
let regressor = classifier.convertToRegression();
let regressor = new ml5.ImageRegressor(classifier);
let regressor = ml5.transferRegression(classifier);
let regressor = classifier.lastLayerTransferLearningRegression(); 😜

I think I like asRegression(). Whatever we do, the key will be the documentation and language around this to be very clear about the difference.

[genekogan]

i think i understand what you mean now. i guess you think of it as a classifier because mobilenet was originally a classifier. but the model that's downloaded has the original classification layer removed, and its last layer just outputs a flattened 7x7x256 tensor (i think probably the last convolution, pre-classification). in ml5, you then attach another layer on the end of that for your custom classification, and hence it becomes (or "re-becomes" depending on how you think about it) a classifier with NUM_CLASSES classes. with regression, the only difference is that the last layer you add is not a classifier, it is a regressor.

so i'm only worried your naming conventions give the impression there is some classifier inside the regressor, which could be misleading. sorry if i seem like i'm splitting hairs! i think ultimately it's fine one way or the other, as long as people don't end up thinking there's a functional classifier inside the regressor.

my suggestion would just be:

let classifier = ml5.ImageClassifier('MobileNet');
let regressor = ml5.ImageRegressor('MobileNet');

but whatever you think is good, i'll happy to go with that!

[genekogan]

or even more radical... you could even split it up in the following way:

let featExtractor = ml5.ImageFeatureExtractor('MobileNet');
let classifier = ml5.ImageClassifier(featExtractor);
let regressor = ml5.ImageRegressor(featExtractor);

where ml5.ImageFeatureExtractor('MobileNet'); just outputs the bare output from the chopped off mobilenet.

an alternative convention could be:

let featExtractor = ml5.ImageFeatureExtractor('MobileNet');
let classifier = featExtractor.asClassifier();
let regressor = featExtractor.asRegressor();

the reason this is nice is that you can do a lot more things with the feature extractor than just classification and regression. you can use the feature vectors to do nearest neighbor image retrieval, you can use them to make a t-SNE, etc... lots of things ;)

[cvalenzuela]

I really like where this is going. I think using the concept of FeatureExtractor is interesting. What makes more sense to me now is to have the FeatureExtractor class that could also be used for other models (not just image?) So, as @genekogan said, something like:

let featExtractor = ml5.FeatureExtractor('MobileNet'); // For images
let regressor = featExtractor.asRegressor();

And then we could work on something like:

let featExtractor = ml5.FeatureExtractor('some-sound-model');
let regressor = featExtractor.asRegressor();

This is also more managble to develop since it can be more modular.

[shiffman]

I'm liking where this is going. @cvalenzuela let's hash this out a bit more in person next week!

[shiffman]

Noting some discussion in #142.

[cvalenzuela]

Just making a note that we ended up settling for this

const features = new ml5.FeatureExtractor('MobileNet');

const classifier = features.classification();
classifier.addImage(img1, 'cat');
classifier.addImage(img2, 'dog');
classifier.train();
classifier.classify(img3, gotResult);

const predictor = features.regression();
predictor.addImage(img1, 0.1);
predictor.addImage(img2, 0.9);
predictor.train();
predictor.predict(img2, gotResult);

This is now implemented (see https://github.com/ml5js/ml5-library/pull/142) and there is a demo in the page here https://ml5js.org/docs/custom-regression

Thanks for the idea and feedback @genekogan!

Closing this as it seems resolved!