ibmtjbot / node-tjbotlib

Node.js library for programming TJBot
https://ibmtjbot.github.io
Apache License 2.0
33 stars 44 forks source link

Implement new Rock Paper Scissors recipe #73

Open modcarroll opened 2 years ago

modcarroll commented 2 years ago

Watson Visual Recognition is deprecated, so we should find a (preferably) open source replacement

jweisz commented 1 year ago

Cesar will own this, he experimented with OpenCV, will train an object detection model for the RPi

jweisz commented 1 year ago

@cmaciel is getting 1.1-1.5 FPS with a pre-trained model (MobileNet, has 90 classes of objects) on a RPi 3. He will re-train the model to focus only on objects found inside a house to get better results.

jweisz commented 1 year ago

Oops, the model was trained but the data labels were missing! :D

jweisz commented 1 year ago

Need to create a list of common household items to train a new model. Maybe focus the model on rock, paper, scissors? Maybe tjbotlib has generic functions for loading local models and running inference, and then we create a separate recipe that uses that API to load a rock/paper/scissors model locally in order to play that game.

jweisz commented 1 year ago

I propose a two-part solution for the elimination of Watson Visual Recognition:

  1. Remove the see() functionality from tjbotlib and add a look() function which just takes a picture and returns the filename in the filesystem (e.g. somewhere in /tmp). Which is to say, I think this is what takePhoto() already does, so maybe we just rename that function.
  2. Add a new recipe to tjbot for rock-paper-scissors that uses @cmaciel 's on-device vision model. But we likely don't want to bundle the whole model inside the tjbot repo because it's likely big (right?) so maybe the recipe downloads the model from somewhere on first run (e.g. test if it's in the filesystem, if not, then download it). We'd need a permalink for hosting the model somewhere.

@cmaciel if this plan makes sense to you, I'd like to close this card and open two new cards to cover 👆 . Thanks!