Plugin not classifying in Phonegap Developers App.

FelixOBrien commented 7 years ago

Hi, I've been trying to create a simple classifying app, with camera functionality with which the image would be taken by the phone and the image data given would be classified by TensorFlow. But this is failing to yield any results, I'm not sure whether it is the Phonegap Developers App as camera functionality fails to work on CLI Build. Here is my code, I've tried using the load function but decided it would be best just to get the app working, best user experience or not. Here is my code:

<!DOCTYPE html>
<html>
  <head>
    <title>Capture Photo</title>

    <script type="text/javascript" charset="utf-8" src="cordova.js"></script>
    <script type="text/javascript" charset="utf-8" src="tensorflow.js"></script>  
    <script type="text/javascript" charset="utf-8">

    var pictureSource;   // picture source
    var destinationType; // sets the format of returned value

    // Wait for device API libraries to load
    //
    document.addEventListener("deviceready",onDeviceReady,false);

    // device APIs are available
    //
    function onDeviceReady() {
        pictureSource=navigator.camera.PictureSourceType;
        destinationType=navigator.camera.DestinationType;
       var tf = new TensorFlow('inception-v1');
        alert("Deviceready");

    }

    // Called when a photo is successfully retrieved
    //
    function onPhotoDataSuccess(imageData) {
      // Uncomment to view the base64-encoded image data
      // console.log(imageData);

      // Get image handle
      //
      var smallImage = document.getElementById('smallImage');

      // Unhide image elements
      //
      smallImage.style.display = 'block';

      // Show the captured photo
      // The in-line CSS rules are used to resize the image
      //
      smallImage.src = "data:image/jpeg;base64," + imageData;
        classify();

    }

    // Called when a photo is successfully retrieved
    //
    function onPhotoURISuccess(imageURI) {
      // Uncomment to view the image file URI
      // console.log(imageURI);

      // Get image handle
      //
      var largeImage = document.getElementById('largeImage');

      // Unhide image elements
      //
      largeImage.style.display = 'block';

      // Show the captured photo
      // The in-line CSS rules are used to resize the image
      //
      largeImage.src = imageURI;
    }

    // A button will call this function
    //
    function capturePhoto() {
      // Take picture using device camera and retrieve image as base64-encoded string
      navigator.camera.getPicture(onPhotoDataSuccess, onFail, { quality: 50,
        destinationType: destinationType.DATA_URL });
    }

    // A button will call this function
    //
    function capturePhotoEdit() {
      // Take picture using device camera, allow edit, and retrieve image as base64-encoded string
      navigator.camera.getPicture(onPhotoDataSuccess, onFail, { quality: 20, allowEdit: true,
        destinationType: destinationType.DATA_URL });
    }

    // A button will call this function
    //
    function getPhoto(source) {
      // Retrieve image file location from specified source
      navigator.camera.getPicture(onPhotoURISuccess, onFail, { quality: 50,
        destinationType: destinationType.FILE_URI,
        sourceType: source });
    }

    // Called if something bad happens.
    //
    function onFail(message) {
      alert('Failed because: ' + message);
    }
    function classify(){
         alert("classifying1");
        tf.classify(imgData).then(function(results) {
                alert('classifying');
                results.forEach(function(result) {
                alert(result.title + " " + result.confidence);
                });
            });    
    }

    </script>
  </head>
  <body>
    <button onclick="capturePhoto();">Capture Photo</button> <br>
    <button onclick="capturePhotoEdit();">Capture Editable Photo</button> <br>
    <button onclick="getPhoto(pictureSource.PHOTOLIBRARY);">From Photo Library</button><br>
    <button onclick="getPhoto(pictureSource.SAVEDPHOTOALBUM);">From Photo Album</button><br>
    <img style="display:none;width:60px;height:60px;" id="smallImage" src="" />
    <img style="display:none;" id="largeImage" src="" />
  </body>
</html>

I did it all in one file for simplicity, If there is anything I have done wrong please alert me to that! Thanks.

sheppard commented 7 years ago

Thanks for trying this out. I converted your example to a fenced code block to make it easier to review. This code is a good start but here are some pointers:

This API requires a native binary file, so it won't work in the PhoneGap Developers App (unless we can convince them to include cordova-plugin-tensorflow as a default plugin).
You don't need to specify tensorflow.js since it will be loaded automatically with the plugin.
Be sure call classify(imageData) and not just classify().

FelixOBrien commented 7 years ago

Thank you! I managed to get everything fixed, and got it working with an inception-v1 model. If you would like I could post my code for others to see, or if you have some working code maybe an example project would be helpful for new users! I love this cordova-plugin, and will try using some of the models I already have on my machine.

FelixOBrien commented 7 years ago

@sheppard I'm back, I have an issue running my custom model, where it always throws the error index.html:1 Uncaught (in promise) Problem reading label file! I have tried re-training, changing the name, changing capital letters and even changing the order of the model to label. I followed the instructions as you gave, step by step and re-did them 2 times. But no matter what it always throws this error. This is my code:

<!DOCTYPE html>
<html>
  <head>
    <title>Capture Photo</title>
      <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>

    <script type="text/javascript" charset="utf-8" src="cordova.js"></script>
    <script type="text/javascript" charset="utf-8">

    var pictureSource;   // picture source
    var destinationType; // sets the format of returned value

    // Wait for device API libraries to load
    //
    document.addEventListener("deviceready",onDeviceReady,false);

    // device APIs are available
    //
    function onDeviceReady() {

        pictureSource=navigator.camera.PictureSourceType;
        destinationType=navigator.camera.DestinationType;
    }

    // Called when a photo is successfully retrieved
    //
    function onPhotoDataSuccess(imageData) {
      // Uncomment to view the base64-encoded image data
      // console.log(imageData);
        var imageData1 = imageData;
      // Get image handle
      //
      var smallImage = document.getElementById('smallImage');

      // Unhide image elements
      //
      smallImage.style.display = 'block';

      // Show the captured photo
      // The in-line CSS rules are used to resize the image
      //
      smallImage.src = "data:image/jpeg;base64," + imageData;
                    var tf = new TensorFlow('custom-model', {
    'label': 'custom-model',
    'model_path': "https://www.dl.dropboxusercontent.com/s/vhnpkhjsjpzmx43/Tensorflow.zip#graph.pb",
    'label_path': "https://www.dl.dropboxusercontent.com/s/vhnpkhjsjpzmx43/Tensorflow.zip#a.txt",
    'input_size': 299,
    'image_mean': 128,
    'image_std': 128,
    'input_name': 'Mul',
    'output_name': 'final_result'
})
            alert('Classifying');
             tf.classify(imageData1).then(function(results) {

    results.forEach(function(result) {
        alert(result.title + " " + result.confidence);
    });
}); 

    }

    // Called when a photo is successfully retrieved
    //
    function onPhotoURISuccess(imageURI) {
      // Uncomment to view the image file URI
      // console.log(imageURI);

      // Get image handle
      //
      var largeImage = document.getElementById('largeImage');

      // Unhide image elements
      //
      largeImage.style.display = 'block';

      // Show the captured photo
      // The in-line CSS rules are used to resize the image
      //
      largeImage.src = imageURI;
    }

    // A button will call this function
    //
    function capturePhoto() {
      // Take picture using device camera and retrieve image as base64-encoded string
      navigator.camera.getPicture(onPhotoDataSuccess, onFail, { quality: 50,
        destinationType: destinationType.DATA_URL });
    }

    // A button will call this function
    //
    function capturePhotoEdit() {
      // Take picture using device camera, allow edit, and retrieve image as base64-encoded string
      navigator.camera.getPicture(onPhotoDataSuccess, onFail, { quality: 20, allowEdit: true,
        destinationType: destinationType.DATA_URL });
    }

    // A button will call this function
    //
    function getPhoto(source) {
      // Retrieve image file location from specified source
      navigator.camera.getPicture(onPhotoURISuccess, onFail, { quality: 50,
        destinationType: destinationType.FILE_URI,
        sourceType: source });
    }

    // Called if something bad happens.
    //
    function onFail(message) {
      alert('Failed because: ' + message);
    }
        function classify(){

        }
        function download(){
var tf = new TensorFlow('custom-model', {
    'label': 'My Custom Model',
    'label_path': "https://www.dl.dropboxusercontent.com/s/vhnpkhjsjpzmx43/Tensorflow.zip#a.txt",
    'model_path': "https://www.dl.dropboxusercontent.com/s/vhnpkhjsjpzmx43/Tensorflow.zip#graph.pb",
    'input_size': 299,
    'image_mean': 128,
    'image_std': 128,
    'input_name': 'Mul',
    'output_name': 'final_result'
});
         tf.checkCached().then(function(isCached) {

    if (isCached) {
        $('#download').hide();
        $('#progress').hide();

    }else{

        tf.onprogress = function(evt) {
    if (evt['status'] == 'downloading'){

        console.log(evt.label);
        if (evt.detail) {
            // evt.detail is from the FileTransfer API
            var elem = document.getElementById('progress');
            var percentage = parseInt(evt.detail.loaded/evt.detail.total * 100);
           console.log(percentage);
            elem.innerHTML = percentage;

        }
    } else if (evt['status'] == 'unzipping') {
        console.log("Extracting contents...");
    } else if (evt['status'] == 'initializing') {
       console.log("Initializing TensorFlow");
    }
}; 

    tf.load().then(function() {
    console.log("Model loaded");
});
    }
         });
                    }

    </script>
  </head>
  <body>
      <button onclick='download();' id = 'download'>Download</button>
      <h1 id = 'progress' name = 'progress'>0%</h1>
    <button onclick="capturePhoto();">Capture Photo</button> <br>
    <button onclick="capturePhotoEdit();">Capture Editable Photo</button> <br>
    <button onclick="getPhoto(pictureSource.PHOTOLIBRARY);">From Photo Library</button><br>
    <button onclick="getPhoto(pictureSource.SAVEDPHOTOALBUM);">From Photo Album</button><br>
    <img style="display:none;width:60px;height:60px;" id="smallImage" src="" />
    <img style="display:none;" id="largeImage" src="" />
  </body>
</html>

Is there anything I'm doing wrong? (The Labels file has been tried starting with Capitals and lowercase, It's currently in capitals from my last attempt) Thanks!

sheppard commented 7 years ago

It looks like the files are in a nested folder within the zipfile. It should work by changing the path to Tensorflow.zip#TensorFlow/a.txt or by moving the files to the top of the zipfile.

FelixOBrien commented 7 years ago

On my end these files are not nested, but maybe the dropbox upload is causing some issues! I'll try that out now. Thanks.

FelixOBrien commented 7 years ago

On my end these files are not nested, but maybe the dropbox upload is causing some issues! I'll try that out now. Thanks.

FelixOBrien commented 7 years ago

Sorry for wasting your time. Completely forgot to check my actual dropbox download file, as I had some weird idea that everything would be alright, but my file address was just completely wrong. Really enjoy using this plugin, Thanks for all the help and work you've put into it, hopefully that's all from me issues wise! If I ever get anything going with it I'll be sure to let you know!

sheppard commented 7 years ago

Great, glad you got it working. Feel free submit a pull request with a simple example project if you get a chance.

heigeo / cordova-plugin-tensorflow

Plugin not classifying in Phonegap Developers App. #2