Tech Analysis: Project Concept

[hwang26]

Frontend interface for ml5.js DoodleNet Classifier

---

https://www.youtube.com/watch?v=jmznx0Q1fP0&ab_channel=TheCodingTrain

https://www.youtube.com/watch?v=3MqJzMvHE3E&ab_channel=TheCodingTrain

https://www.youtube.com/watch?v=ABN_DWnM5GQ&ab_channel=TheCodingTrain

• [x] Watch videos for background

---

• [x] Attach notes

---

• user login/auth
• Disclaimer on the model (DoodleNet)
• Pane for mouse controlled drawing
  • We could make this a game where we want to get within 90% range of accuracy for a given category
• Classification information
  • we can be clever with this by using a format string
• Clear button
• Save button
• past 5 saved images
• ability to click on an image to bring it up in the classifier as a background
  • allow user to draw over the image

---

[hwang26]

https://github.com/hwang26/image-classifier/issues/1

---

A Beginner's Guide to Machine Learning with ml5.js

https://www.youtube.com/watch?v=jmznx0Q1fP0&ab_channel=TheCodingTrain

---

• NYU Red Burns

https://itp.nyu.edu/adjacent/issue-3/ml5-friendly-open-source-machine-learning-library-for-the-web/

---

• p5js

  • library

  • setup(), draw()

• ml5js

  • library

  • image classifier

  • uses tfjs

  • the "5" is a tribute to "p5"

• tfjs, keras

  • python version compiles to C++

  • tfjs runs all in js

  • lower level library than ml5js

    • eg. memory management

---

• algorithms

  • eg. neural network

• datasets

• models

  • eg. pretrained model

  • MobileNet

[x] Maybe include a tooltip on how our pretrained model works and what dataset it uses

- For transparency reasons

---

https://www.youtube.com/watch?v=3MqJzMvHE3E&ab_channel=TheCodingTrain

Coding Challenge #158: Shape Classifier Neural Network with ml5.js

---

• Manually trained CNN that guesses shapes

---

Generate Model Dataset

• setup()

• draw()

  • Drawing circles

    • saveFrame()

      • saves pngs

      • save 100 frames

  • draw square, triangle

  • use rotate() and random() to change up alignment so that they're not all perfectly aligned

  • rectmode(center)

    • keep an idea of the camera view when we're drawing

[-] Should we use a pretrained model or should we use our own datasets?

- We could provide some popular datsets that can be switched in and out on the frontend?

- Let's just use DoodleNet and keep it simple for now

---

• preload()

  • preload all the images as arrays of circles, squares, triangles

• setup()

  • draw images to make sure its working

• see in browser?

  • run local http-server in commandline

• create neuralnetwork

  shapeClassifier = ml5.neuralNetwork(options)

• shape of data?

  • size of image

    128 x 128

  • channels of image (colors, alpha value, greyscale)

  • specify task as CNN

    • 'imageClassification'

  • set debug to true

    • automatic visualization of loss function

    • shows us the error over time during training

• add data for circle, squares, triangle

  • training dataset has "input" and "target"

  • shapeClassifier.addData(iinput, target)

• normalize data

  • analyze data, look at min and max ranges of pixel values, normalize all inputs to numbers between 0 and 1

• train network

  • shapeClassifier.train({epochs: 50}, finishedTraining)

    • send all data 50 times through the neural network

  • callback finishedTraining is just a console.log()

[-] Training takes a while, if we don't do preprocessing, we may want to include a loading bar or icon

- We won't need to do this bc we can just upload the pretrained model

---

Training is not working super well

• reduce resolution

• increase stroke weight

• reduce random() so the images don't bleed off the edge

• reduce input size of image for classifier options

  64x64

---

• No way to reverse training data from neural network itself

• Upload the pretrained model

p5 sketch

• set up sketch and model

• shapeClassifier.load(modelDetails, modelLoaded)

  • modelLoaded() is just a callback that runs console.log()

• pass an image to NN

  • test w/draw()

• mousePressed()

  • create input graphics object

  • input.copy(..)

  • image(..)

• want to constantly classify drawn image

• want to see the results in the page

• clear button

---

https://www.youtube.com/watch?v=ABN_DWnM5GQ&t=229s&ab_channel=TheCodingTrain

ml5.js: Classifying Drawings with DoodleNet

---

• Pretrained model DoodleNet

  • 345 categories of sketches

---

• doodleClassifier = ml5.imageClassifier('DoodleNet', modelReady)

• get model ready

• classify the canvas

• ml5 error first callback pattern

• returns objects with classification ideas

---

• cat drawing is not working

• guess: dataset is consistent

  • eg. line thickness

• increase stroke weight

---