How to use base64 data with tensorflow?

[AnthonyNTK]

Is there any way to use base64 string as the input for tensorflow image api or any workaround like sort of conversion of data type?

I saw #9 and #2 but didn't get the idea to do such conversion.... Any concrete example can be provided? Thanks in advance.

[reneweb]

Currently the image api does not support passing the image as base64. It should not be too difficult to add this support although it is generally not recommended to pass large payloads like images between js and native - maybe I will add this functionality though anyway at some point.

The second option would be to use the direct api, but that would require to convert the base64 string into a format that can be processed by TF. I don't have any js examples for this unfortunately and this won't be an easy thing to do - you would need to do some form of conversion like this on the js side: https://github.com/reneweb/react-native-tensorflow/blob/master/android/src/main/java/com/rntensorflow/imagerecognition/ImageRecognizer.java#L71-L85

The third option would be to write your base64 string to a file using react-native-fs like so (https://github.com/itinance/react-native-fs#writefilefilepath-string-contents-string-encoding-string-promisevoid) and then use the image recognition api referencing this file - this is probably your best bet to get this working for now

[AnthonyNTK]

Thanks for your suggestion and I can use base64 string to do the recognition jobs now. Everything is good while the app was running on a physical device using react-native run-android, but the app would prompt "Could not fetch data from url https://localhost:8081/assets/assets/tensorflow_inception_graph.pb?..... "

so i guess react native engine can't recognize the path as expected? Any idea?

[sparkison]

@AnthonyNTK, do you mind letting me know how you managed to make it work with base64 images? I'm running into an issue where I'm not able to use require and have a path to the file in the documents directory instead and passing the path string to the TF recognize function isn't working

[AnthonyNTK]

Situation may differ on how you get your image. For me, I tried to implement a function that user can take photo as the source for tensorflow engine to recognize it. So I used react-native-camera which supports returning a base64 string and specifying the quality (the length of base74 string) of the image taken. On the other hand, I will save that base64 string to a hard-coded destination in the device by using react-native-fs (either cache or asset file or anywhere you prefer). So here is the trick, which i believe @reneweb suggests me to do, by pointing the source of image of the tensorflow engine to the directory where you save the image (Also in a hard-coded way), for me it is ${RNFS.DocumentDirectoryPath}/temp.png. This is definitely not a generic way to solve the issue, but for me, I would like to just take some hands-on experience to the AI kits.

[sparkison]

Thanks @AnthonyNTK!

So you're just doing something like this then?

const results = await tfImageRecognition.recognize({
  image: `${RNFS.DocumentDirectoryPath}/temp.png`,
})

as apposed to this

const results = await tfImageRecognition.recognize({
  image: require('./assets/temp.jpg'),
})

[reneweb]

@sparkison yeah, it should look like the first one. require only works if the asset is actually bundled with the app - in this case you want to read it from file system.

[sparkison]

@reneweb thank you! I'll give it a shot. I tried passing the path and it didn't seem to work, I'll try it again and see what I get...

[sparkison]

Hey @reneweb, still not able to load in a file using the path, something like file:///var/mobile/Containers/Data/Application/89317E57-4EFF-49F4-90DC-62588E435374/Documents/sample_6.jpg

When setting breakpoints in the ImageRecognizer.mm file where the check is, around line 56, this check returns false: ...else if ([[NSFileManager defaultManager] fileExistsAtPath:image])..., which I would think should return true? Because of this the image is loaded using this line in the same file at line 62: imageData = [[NSData alloc] initWithContentsOfURL: imageUrl];, which seems to output false/random results.

Any help appreciated!!

[reneweb]

Yeah, the call [[NSFileManager defaultManager] fileExistsAtPath:image] should return true of the file exists in the document directory. I tried myself to just add an image to the document directory in the emulator and it did return true for me when running this check. Maybe worth double checking if that file does indeed exist at that location (printing out the contents of the document directory might help).

[sparkison]

Got it! It was the file:// I prepended to the path. Other libraries I've used prefer this, apparently the fileExistsAtPath method just expects the full path.

Sorry to spam this thread! And thank you for taking the time to help troubleshoot ;)

[reneweb]

Cool - I found a similar issue on Android version. I will probably add something on both Android and iOS to remote the file prefix if existing

[vaibhgupta09]

@reneweb @sparkison guyz can I do real time object detection with this library?? I.e it will show result whenever camera detect objects.

[chirag773]

hey @reneweb right now we can only pass the image statically through assets folder but i want to pass the photo dynamically. I'm using react-native-image-picker for taking the images and when I'm passing that image to the tfImageRecognition.recognize .

 const results = await tfImageRecognition.recognize({

        image: this.state.pickedImage
      })

it is showing me an error could not load the resources can you help me out.

[lutfimaulana17]

hey @reneweb right now we can only pass the image statically through assets folder but i want to pass the photo dynamically. I'm using react-native-image-picker for taking the images and when I'm passing that image to the tfImageRecognition.recognize .

 const results = await tfImageRecognition.recognize({

        image: this.state.pickedImage
      })

it is showing me an error could not load the resources can you help me out.

i'm same this issue

[flyskywhy]

I just create react-native-get-asset which described how to bundle asset automatically at bundle time, and one function to copy asset to file system or read asset as base64 data url on Android, iOS and Web at debug or release run time. :stuck_out_tongue: