Simple Non-Image Based Example

[bvallelunga]

Hey @mats-claassen thank you for building this amazing library! This is exactly what I was looking for!!!

I noticed you have a lot of examples with Image based ML and am struggling to get started with a simple non-image example. I have a model built in tensorflow that is 1 layer using Sigmoid that takes in 20 features and outputs 1 label, Binary Classification. I am struggling to get my model to run successfully on Bender. Here are the errors I am seeing.

2017-06-29 17:48:17.286368-0700 bender_test[409:138318] libMobileGestalt MobileGestaltSupport.m:153: pid 409 (bender_test) does not have sandbox access for frZQaeyWLUvLjeuEK43hmg and IS NOT appropriately entitled
2017-06-29 17:48:17.286403-0700 bender_test[409:138318] libMobileGestalt MobileGestalt.c:550: no access to InverseDeviceID (see <rdar://problem/11744455>)
2017-06-29 17:48:17.391311-0700 bender_test[409:138318] /BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-61.5/MetalPerformanceShaders/Filters/MPSKernel.mm, line 370: error 'Source 0x101118830 texture type must be MTLTextureType2D
'
/BuildRoot/Library/Caches/com.apple.xbs/Sources/MetalImage/MetalImage-61.5/MetalPerformanceShaders/Filters/MPSKernel.mm:370: failed assertion `Source 0x101118830 texture type must be MTLTextureType2D'

Do you have an example I can look at where the input MPSImage is composed of a 20 feature tensor?

[johndpope]

not what you were looking for - but distantly related albeit this is for server side stuff. swift + tensorflow + custom models. https://github.com/PerfectExamples/Perfect-TensorFlow-Demo-Vision

[bvallelunga]

@johndpope do you know if it provides iOS support and if the library size doesn't increase the apps binary size to much? Im capped at increasing our apps size by 5mb.

[bvallelunga]

@dernster any thoughts on how to run a simple non-image predict on Bender from a TensorFlow file?

[johndpope]

hi @bvallelunga - no ios support. the native c++ tensorflow stuff out of the box does have support for ios - as of the other week they now have a development cocoapod of 500mb. not sure how this impacts final build size. I've found it a challenge to do the most basic loading of custom models that are not image based though.

It's likely if there's demand that the swift wrapper that @RockfordWei created for desktop tensorflow could be adopted to fit ios down the track. though i suspect that there's no immediate plans. there's a complicated stripping and scaling down that happens when you build for ios, not all ops are supported.

[bvallelunga]

@johndpope that interesting, though 500mb is like crazy large. I am also having trouble getting a very basic model to work on bender. I am getting these errors:

"\n\n\nNodes in my graph (pre optimization):"
"inputs/x-input"
"model/W"
"model/MatMul"
"model/b"
"model/add"
"model/y_pred"
"inference/Greater/y"
"inference/Greater"
"inference/inference"
"\n\n\nNodes in my graph (after optimization):"
"inputs/x-input"
"model/W"
"model/b"
"model/add"
"inference/Greater/y"
"inference/Greater"
"inference/inference"
"Bender:: Unsupported layer found: Placeholder"
"Bender:: Unsupported layer found: Const"
"Bender:: Unsupported layer found: Const"
"Bender:: Unsupported layer found: Const"
"Bender:: Unsupported layer found: Greater"
"Bender:: Unsupported layer found: Cast"

This is the TensorFlow python script I used to generate the model:

# This script is used to train the model. It repeats indefinitely and saves the
# model every so often to a checkpoint. 
#
# Press Ctrl+C when you feel that training has gone on long enough (since this is 
# only a simple model it takes less than a minute to train, but a training a deep l
# earning model could take days).

import os
import numpy as np
import tensorflow as tf

checkpoint_dir = "/tmp/voice/"
print_every = 1000
save_every = 10000
num_inputs = 20
num_classes = 1

# Load the training data.
X_train = np.load("X_train.npy")
y_train = np.load("y_train.npy")

print("Training set size:", X_train.shape)

# Below we'll define the computational graph using TensorFlow. The different parts 
# of the model are grouped into different "scopes", making it easier to understand
# what each part is doing.

# Hyperparameters let you configure the model and how it is trained. They're
# called "hyper" parameters because unlike the regular parameters they are not
# learned by the model -- you have to set them to appropriate values yourself.
#
# The learning_rate tells the optimizer how big of a steps it should take.
# Regularization is used to prevent overfitting on the training set.
with tf.name_scope("hyperparameters"):
    regularization = tf.placeholder(tf.float32, name="regularization")
    learning_rate = tf.placeholder(tf.float32, name="learning-rate")

# This is where we feed the training data (and later the test data) into the model. 
# In this dataset there are 20 features, so x is a matrix with 20 columns. Its number 
# of rows is None because it depends on how many examples at a time we put into this 
# matrix. This is a binary classifier so for every training example, y gives a single 
# output: 1 = male, 0 = female.
with tf.name_scope("inputs"):
    x = tf.placeholder(tf.float32, [None, num_inputs], name="x-input")
    y = tf.placeholder(tf.float32, [None, num_classes], name="y-input")

# The parameters that we'll learn consist of W, a weight matrix, and b, a vector
# of bias values. (Actually, b is just a single value since the classifier has only
# one output. For a classifier that can recognize multiple classes, b would have as
# many elements as there are classes.)
with tf.name_scope("model"):
    W = tf.Variable(tf.zeros([num_inputs, num_classes]), name="W")
    b = tf.Variable(tf.zeros([num_classes]), name="b")

    # The output is the probability the speaker is male. If this is greater than
    # 0.5, we consider the speaker to be male, otherwise female.
    y_pred = tf.sigmoid(tf.matmul(x, W) + b, name="y_pred")

# This is a logistic classifier, so the loss function is the logistic loss.
with tf.name_scope("loss-function"):
    loss = tf.losses.log_loss(labels=y, predictions=y_pred)

    # Add L2 regularization to the loss.
    loss += regularization * tf.nn.l2_loss(W)

# Use the ADAM optimizer to minimize the loss.
with tf.name_scope("train"):
    optimizer = tf.train.AdamOptimizer(learning_rate)
    train_op = optimizer.minimize(loss)

# For doing inference on new data for which we don't have labels.
with tf.name_scope("inference"):
    inference = tf.to_float(y_pred > 0.5, name="inference")

# The accuracy operation computes the % correct on a dataset with known labels. 
with tf.name_scope("score"):
    correct_prediction = tf.equal(inference, y)
    accuracy = tf.reduce_mean(tf.to_float(correct_prediction), name="accuracy")

init = tf.global_variables_initializer()

# For writing training checkpoints and reading them back in.
saver = tf.train.Saver()
tf.gfile.MakeDirs(checkpoint_dir)

with tf.Session() as sess:
    # Write the graph definition to a file. We'll load this in the test.py script.
    tf.train.write_graph(sess.graph_def, checkpoint_dir, "graph.pb", False)

    # Reset W and b to zero.
    sess.run(init)

    # Sanity check: the initial loss should be 0.693146, which is -ln(0.5).
    loss_value = sess.run(loss, feed_dict={x: X_train, y: y_train, regularization: 0})
    print("Initial loss:", loss_value)

    # Loop forever:
    step = 0
    while True:
        # We randomly shuffle the examples every time we train.
        perm = np.arange(len(X_train))
        np.random.shuffle(perm)
        X_train = X_train[perm]
        y_train = y_train[perm]

        # Run the optimizer over the entire training set at once. For larger datasets
        # you would train in batches of 100-1000 examples instead of the entire thing.
        feed = {x: X_train, y: y_train, learning_rate: 1e-2, regularization: 1e-5}
        sess.run(train_op, feed_dict=feed)

        # Print the loss once every so many steps. Because of the regularization, 
        # at some point the loss won't become smaller anymore. At that point, it's
        # safe to press Ctrl+C to stop the training.
        if step % print_every == 0:
            train_accuracy, loss_value = sess.run([accuracy, loss], feed_dict=feed)
            print("step: %4d, loss: %.4f, training accuracy: %.4f" % \
                    (step, loss_value, train_accuracy))

        step += 1

        # Save the model. You should only press Ctrl+C after you see this message.
        if step % save_every == 0:
            checkpoint_file = os.path.join(checkpoint_dir, "model")
            saver.save(sess, checkpoint_file)            
            print("*** SAVED MODEL ***")

@dernster or @mats-claassen any thoughts?

[dernster]

@bvallelunga Thanks. You can take a look at this test. Here we use an internal Texture class that can be created from arrays, then it's converted to a MTLTexture and finally to a MPSImage. In your particular case I see that your model has a "Greater" layer and we don't have support for it at this moment. We are open to receive pull request though :)

[bvallelunga]

@dernster Thank you! Yea I just noticed you don't support the Greater layer. Do you plan on adding support for it soon?

Also does code look good for building a MPSImage?

extension MPSImage {
   convenience init(input: [Float], device: MTLDevice) {
        // The input data must be placed into the MTLTexture from the MPSImage.
        // First we convert it to 16-bit floating point, then copy these floats
        // into the texture slices. (Note: this will fail if the number of channels
        // is not a multiple of 4 -- in that case you should add padding bytes to
        // the example array.)
        var input = input
        let input16 = float32to16(&input, count: input.count)
        let inputImgDesc = MPSImageDescriptor(channelFormat: .float16, width: 1, height: 1, featureChannels: input.count)
        self.init(device: device, imageDescriptor: inputImgDesc)
        input16.withUnsafeBufferPointer { ptr in
            for i in 0..<self.texture.arrayLength {
                let region = MTLRegion(origin: MTLOriginMake(0, 0, 0), size: MTLSizeMake(1, 1, 1))
                self.texture.replace(region: region, mipmapLevel: 0, slice: i, withBytes: ptr.baseAddress!.advanced(by: i*4), bytesPerRow: MemoryLayout<Float16>.stride * 4, bytesPerImage: 0)
            }
        }
    }
}

[bryant1410]

@bvallelunga there's no plan to add it soon. But maybe you can try to implement it yourself, by taking a look at some other layer!

[dernster]

@bvallelunga Not sure but you can see our Texture class. It already has that functionality and without the multiple of 4 restriction. Maybe we would need to add it to Bender, since right now it's at the Example project.

[bvallelunga]

@dernster would you be able to give me an example of using the functionality without the multiple of 4 restriction?

[RockfordWei]

Thanks, @johndpope for the comment. For TensorFlow iOS solution, however, I think CoreML maybe a better approach - you can import TensorFlow or even Keras models to CoreML. That's one of the most important reasons that I am not going to migrate Python core code into Swift right now without our sponsors' permission. Sorry about that.

[bryant1410]

Thanks, @johndpope for the comment. For TensorFlow iOS solution, however, I think CoreML maybe a better approach - you can import TensorFlow or even Keras models to CoreML. That's one of the most important reasons that I am not going to migrate Python core code into Swift right now without our sponsors' permission. Sorry about that.

Take into account that Core ML doesn't support TensorFlow without Keras, it only supports versions 1.0.X and 1.1.x and it doesn't support adding custom layeers. Also, it works with iOS 11+, which hasn't been released.

[RockfordWei]

If anyone can be the sponsor I think I can translate the whole TensorFlow python library into Swift

On Fri, Jun 30, 2017 at 17:53 Santiago Castro notifications@github.com wrote:

Thanks, @johndpope https://github.com/johndpope for the comment. For TensorFlow iOS solution, however, I think CoreML maybe a better approach - you can import TensorFlow or even Keras models to CoreML. That's one of the most important reasons that I am not going to migrate Python core code into Swift right now without our sponsors' permission. Sorry about that.

Take into account that Core ML doesn't support TensorFlow without Keras, it only supports versions 1.0.X and 1.1.x and it doesn't support adding custom layeers. Also, it works with iOS 11+, which hasn't been released.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xmartlabs/Bender/issues/36#issuecomment-312380985, or mute the thread https://github.com/notifications/unsubscribe-auth/AVIpO_TisRZQhyE2LlQRGf3fPnQNwoArks5sJW5PgaJpZM4OKCfL .

[bvallelunga]

@RockfordWei would the the completed framework support iOS and be lightweight (under 2mb)?

[RockfordWei]

On Fri, Jun 30, 2017 at 17:57 Brian Vallelunga notifications@github.com wrote:

@RockfordWei https://github.com/rockfordwei would the the completed framework support iOS and be lightweight (under 2mb)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xmartlabs/Bender/issues/36#issuecomment-312381615, or mute the thread https://github.com/notifications/unsubscribe-auth/AVIpOxE3oH6liZySaaosqXOvwRYQlI18ks5sJW9LgaJpZM4OKCfL .

No guarantee, I didn't poke it much

[bvallelunga]

@RockfordWei what kind of sponsorship are you looking for?

[RockfordWei]

On Fri, Jun 30, 2017 at 18:08 Brian Vallelunga notifications@github.com wrote:

@RockfordWei https://github.com/rockfordwei what kind of sponsorship are you looking for?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xmartlabs/Bender/issues/36#issuecomment-312383434, or mute the thread https://github.com/notifications/unsubscribe-auth/AVIpO0EfBNeoZvXgXazbKT41XU_s-E2nks5sJXHXgaJpZM4OKCfL .

See perfect.org

[bvallelunga]

@RockfordWei I have checked out the site, is there a specific page you would like me to look at?

[RockfordWei]

On Fri, Jun 30, 2017 at 18:13 Rocky WEI rockywei524@gmail.com wrote:

On Fri, Jun 30, 2017 at 18:08 Brian Vallelunga notifications@github.com wrote:

@RockfordWei https://github.com/rockfordwei what kind of sponsorship are you looking for?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xmartlabs/Bender/issues/36#issuecomment-312383434, or mute the thread https://github.com/notifications/unsubscribe-auth/AVIpO0EfBNeoZvXgXazbKT41XU_s-E2nks5sJXHXgaJpZM4OKCfL .

See perfect.org

We are attracting the third round of investment, actually

[bvallelunga]

Ahhh, no thank you. I am just a single developer working on a side project.

[dernster]

@bvallelunga Sorry for the delay, here's an example:

Create your texture:

let texture = Texture(
            data: [
                [1.0, 2.0, 3.0, 4.0, 5.0, 6], [9.0, 10.0, 11.0, 12.0, 13.0, 14],
                [17.0, 18, 19, 20, 21, 22], [25, 26, 27, 28, 29, 30],
                [33, 34, 35, 36, 37, 38], [41, 42, 43, 44, 45, 46]
            ],
            size: LayerSize(f: 6, w: 2, h: 3)
)

Here, each array corresponds to z-values. Then, you need to create a metal texture:

let metalTexture = texture.metalTexture(with: device)

And finally a MPSImage instance:

let mpsImage = MPSImage(texture: metalTexture, featureChannels: texture.depth)

After your network returns you can create a computedTexture from the MPSImage returned:

let computedTexture = Texture(metalTexture: returnedImage.texture, size: /* Layer size of returned texture */)

Take into account that the class Texture isn't included in Bender, for now it's only on the Examples. Maybe you can just copy that class.