Questions

[Abner77]

Hi, my apologies if this is not the best way to request some help about brightwire. I was trying to use the api and I sstumble on some, probably trivial problems.....

1. The first one would be, how to declare the AlgebraProvider to be gpu tuned, or the intel cpu tuned The Interface of these algebra providers is not BrightWire.ILinearAlgebraProvider, and when calling the constructor for GraphFactory it requieres an BrightWire.ILinearAlgebraProvider. I haven't found an example of this....

2. When creating an engine, the graph requires the datatable already formed. What would be the best way to go if I want to implement something like a Reinforcement Learning algorithm in which I have to play some matches at some game, store the data, apply training, repeat.... The samples I've seen assume an static set of samples. Is it good practice to recreate continuosly the engine? Does that affect the performance or the weights already trained?

3 Once I have the model trained, how do I use the neural network or graph for online inference?

4. Is there a way to save and load the model trained?

Thank you for your fantastic work. It seems like a good Machine learning library for .net

[jdermody]

Hi

Take a look at SampleCode/MNIST.cs

1) You create a linear algebra provider with BrightWireGpuProvider.CreateLinearAlgebra() or BrightWireProvider.CreateLinearAlgebra() that will execute on the GPU or CPU respectively.

2,3,4) The execution graph with the trained weights is a protobuf object that is serialisable. Take a look at BrightWire.Source/Models/ExecutionGraph.cs and https://github.com/mgravell/protobuf-net

At the end of the MNIST sample is an example of loading an execution engine with an execution graph that was exported by the training engine (in an online setting the graph would have been deserialised from somewhere with the weights gained from training).

There is also a method on IGraphTrainingEngine called LoadParamatersFrom that restores weights from a previous execution graph. So you might complete one round of training, persist the graph somewhere, set up a second round of training and restart with that previous set of learnt parameters.

Let me know if that helps (I'm not sure I answered your second question fully).

[Abner77]

Thank you for your answer! As soon as I can I will try to implement a simple Reinforcement Learning to see what kind of barriers I might find. It's a shame that Microsoft, in 2018 still does not have something like this... Most people has completely ditched c# for machine learning and has moved to python and tensorflow.

[jdermody]

Yeah I think it's a shame too. I think c# makes a lot of sense for ML, especially when deploying ML to production. Maybe as it becomes more mainstream and less tied to research teams it will start to shift over...

[Abner77]

Hi Jack, so here is the problem I find, this is some toy code I have written. I have commented in capital letters where I find the problem to implement an RL learning with BrightWire. Basically speaking, I would need the training data not be attached to a datasource that is attached to an engine which is attached to the graph when creating the neural network. I need to refresh the training data after each call to engine.Train... Is this possible?

//create the data, two sets of data simulating n sets as we would use in reinforcement learning (creating data online) StringBuilder sb = new StringBuilder(); StringBuilder sb2 = new StringBuilder();

        sb.AppendLine("0,0,0,0");
        sb.AppendLine("0,0,1,1");
        sb.AppendLine("0,1,0,1");
        sb.AppendLine("0,1,1,0");            

        //data set 1
        BrightWire.IDataTable tablaDatos = BrightWire.BrightWireProvider.ParseCSV(sb.ToString());

        sb2.Clear();
        sb2.AppendLine("1,0,0,1");
        sb2.AppendLine("1,0,1,0");
        sb2.AppendLine("1,1,0,0");
        sb2.AppendLine("1,1,1,1");

        //data set 2
        BrightWire.IDataTable tablaDatos2 = BrightWire.BrightWireProvider.ParseCSV(sb2.ToString());

        // the last column is the classification target ("Iris-setosa", "Iris-versicolor", or "Iris-virginica")
        //le indicamos que el valor de salida está en el targetcolumnindex último
        var targetColumnIndex = tablaDatos.TargetColumnIndex = tablaDatos.ColumnCount - 1;
        var targetColumnIndex2 = tablaDatos2.TargetColumnIndex = tablaDatos2.ColumnCount - 1;

        //Create Data sources.
        var trainingData = graph.CreateDataSource(tablaDatos);
        var trainingData2 = graph.CreateDataSource(tablaDatos2);
        //Cçreamos el motor de entrenamiento, donde se le indican los datos a entrenar el training rate y el tamaño del batch

        //TROUBLE HERE
        BrightWire.IGraphTrainingEngine engine = graph.CreateTrainingEngine(trainingData, TRAINING_RATE, 128); 
        BrightWire.IGraphTrainingEngine engine2 = graph.CreateTrainingEngine(trainingData2, TRAINING_RATE, 128);
        //esto sirve para ajustar el training rate cada cierto número de batchs, no lo uso ahora mismo.
        engine.LearningContext.ScheduleLearningRate(100, TRAINING_RATE / 2);
        engine2.LearningContext.ScheduleLearningRate(100, TRAINING_RATE / 2);

        //métrica de error o función de pérdida, cuadrática (mse supongo en el estándar de keras)
        var errorMetric = graph.ErrorMetric.Quadratic; 

        // create the network
        graph.Connect(engine)  //SPECIFIC TROUBLE HERE, THE ENGINE, WHICH HAS ASSOCIATED TRAINING DATA IS SPECIFIED CREATING THE NEURAL NETWORK
            .AddFeedForward(outputSize: 4)  //tipo de capa
            .Add(graph.ReluActivation()) // tipo de activación.  
            .AddFeedForward(outputSize: trainingData.OutputSize)   //última capa, indicamos como tamaño de salida el del trainingdata
            .Add(graph.ReluActivation())
            .AddBackpropagation(errorMetric)
        ;

        // train the network                         
        for (int i = 0; i < 1000; i++)
        { 
            //TROUBLE HERE, I WOULD NEED TO CHANGE THE TRAINING DATA IN EEACH CALL TO TRAIN IS THERE ANYWAY TO DO THAT?
            System.Diagnostics.Debug.WriteLine (engine.Train(context).ToString()); //la salida de esto parece ser el error total (loss)                            
            System.Diagnostics.Debug.WriteLine(engine2.Train(context).ToString()); //la salida de esto parece ser el error total (loss)            
        }

[jdermody]

Apologies for the delay in replying. The simplest way that comes to mind is to create your own data source (a class that implements IDataSource). You could then use your custom object instead of the one returned by graph.CreateDataSource. See the vector based one below as reference, but presumably for RL you would change the array of vectors on the fly as needed:

class VectorDataSource : IDataSource
{
    readonly int _inputSize, _outputSize;
    readonly IReadOnlyList<FloatVector> _data;
    readonly ILinearAlgebraProvider _lap;

    public VectorDataSource(ILinearAlgebraProvider lap, IReadOnlyList<FloatVector> data)
    {
        _lap = lap;
        _data = data;

        var first = data.First();
        _inputSize = first.Size;
        _outputSize = -1;
    }

    public int InputCount => 1;
    public bool IsSequential => false;
    public int InputSize => _inputSize;
    public int OutputSize => _outputSize;
    public int RowCount => _data.Count;

    public IMiniBatch Get(IExecutionContext executionContext, IReadOnlyList<int> rows)
    {
        var data = rows.Select(i => _data[i]).ToList();
        var input = _lap.CreateMatrix(data.Count, InputSize, (x, y) => data[x].Data[y]);
        var inputList = new List<IGraphData> {
            new MatrixGraphData(input)
        };
        return new MiniBatch(rows, this, inputList, null);
    }

    public IReadOnlyList<IReadOnlyList<int>> GetBuckets()
    {
        return new[] {
            Enumerable.Range(0, _data.Count).ToList()
        };
    }

    public IDataSource CloneWith(IDataTable dataTable)
    {
        throw new NotImplementedException();
    }

    public void OnBatchProcessed(IContext context)
    {
        // nop
    }
}

[Abner77]

Thank you for your answer and your time jack. I see, I'll try that as soon as I can. I'm looking for a way to use the cartpole minigame from open ai gym in some way from C#. Let's see if I can put together a functional sample.

Cheers!

[Abner77]

Hi Jack. Sorry I'm back with more problems. I was trying to implement the IDataSource interface following the example you gave me before, but I have a compilation error. The MiniBatch and the MatrixGraphData class you invoke (new MiniBatch), raises an error in compilation time that says that they are not accesible due to their level of protection.... What do I do? Thank you very much for your help

[jdermody]

That's okay, and how annoying! I'll go over the non public classes in the next release and see why they're not public and if they can be exposed.

In the meantime my suggestion would be to add the non public classes as child classes to your RL data source (copy and paste), They're both relatively simple adapter classes just to make things easier.

Please let me know how you go1

[Abner77]

Hi Jack, I've created the DataSource but something does not work. I get a null reference error when I call the train method.:

en BrightWire.ExecutionGraph.Action.Backpropagate.Execute(IGraphData input, IContext context)\r\n en BrightWire.ExecutionGraph.Node.Helper.ExecuteForwardAction.ExecuteForward(IContext context)\r\n en BrightWire.ExecutionGraph.Node.NodeBase.ExecuteForward(IContext context, Int32 channel)\r\n en BrightWire.ExecutionGraph.Engine.Helper.TrainingEngineContext.ExecuteNext()\r\n en BrightWire.ExecutionGraph.Engine.TrainingEngine._Train(IExecutionContext executionContext, ILearningContext learningContext, IMiniBatchSequence sequence)\r\n en BrightWire.ExecutionGraph.Engine.TrainingEngine._Train(IExecutionContext executionContext, ILearningContext learningContext, IMiniBatch batch)\r\n en BrightWire.ExecutionGraph.Engine.TrainingEngine.<>c__DisplayClass13_0.b__0(IMiniBatch batch)\r\n en BrightWire.ExecutionGraph.Helper.MiniBatchProvider.MiniBatchOperation.Execute(IExecutionContext executionContext)\r\n en BrightWire.ExecutionGraph.Engine.TrainingEngine.Train(IExecutionContext executionContext, Action`1 batchCompleteCallback)\r\n en PruebaBrightWire.Form1.CrearModelo() en C:\Users\EXCC0176\source\repos\PruebaBrightWire\PruebaBrightWire\Form1.cs:línea

This is the class that implements IDataSource that I created, I haven't modified it too much, I just wanted to grasp how it works before changing the way it returns the minibatches, etc.

public class Datos : IDataSource { readonly int _inputSize, _outputSize; readonly IReadOnlyList _data; readonly ILinearAlgebraProvider _lap;

    public Datos(ILinearAlgebraProvider lap, IReadOnlyList<FloatVector> data)
    {
        _lap = lap;
        _data = data;

        var first = data.First();
        _inputSize = first.Size - 1;
        _outputSize = 1; //??? 
    }

    public int InputCount => 1;
    public bool IsSequential => false;
    public int InputSize => _inputSize;
    public int OutputSize => _outputSize;
    public int RowCount => _data.Count;

    public IMiniBatch Get(IExecutionContext executionContext, IReadOnlyList<int> rows)
    {
        var data = rows.Select(i => _data[i]).ToList();
        var input = _lap.CreateMatrix(data.Count, InputSize, (x, y) => data[x].Data[y]);
        var inputList = new List<IGraphData> {new MatrixGraphData(input)};

        return new MiniBatch(rows, this, inputList, null);
    }

    public IReadOnlyList<IReadOnlyList<int>> GetBuckets()
    {
        return new[] {Enumerable.Range(0, _data.Count).ToList()};
    }

    public IDataSource CloneWith(IDataTable dataTable)
    {
        throw new NotImplementedException();
    }

    public void OnBatchProcessed(IContext context)
    {
        // nop
    }
}

And this is the method where I create the model and create simple data.....

private GraphFactory CrearModelo() {

        //Creación del proveedor de algebra
        //Con gpu sería , (pero para ello entiendo que el paquete ha de ser el espeacial de cuda (está en el nuget)
        //También hay otro de intel, teóricamente, pero no sé cómo se puede invocar.
        //BrightWire.brightWireGPUProvider
        var lap = BrightWire.BrightWireProvider.CreateLinearAlgebra();            
        //Creamos el graph factory
        GraphFactory graph = new GraphFactory(lap);
        //El contexto, esto es necesario por alguna razón para más tarde.
        var context = graph.CreateExecutionContext(); 

        //Indicamos que el algoritmo de backpropagation será el adamoptimizar
        graph.CurrentPropertySet
        .Use(graph.Adam());

        //rate de entrenamiento
        const float TRAINING_RATE = 0.5f;

        //Creamos los datos (XOR clásico)
        StringBuilder sb = new StringBuilder();            

        sb.AppendLine("0,0,0,0");
        sb.AppendLine("0,0,1,1");
        sb.AppendLine("0,1,0,1");
        sb.AppendLine("0,1,1,0");
        sb.AppendLine("1,0,0,1");
        sb.AppendLine("1,0,1,0");
        sb.AppendLine("1,1,0,0");
        sb.AppendLine("1,1,1,1");

        List<BrightWire.Models.FloatVector> lista = CrearDatos(sb.ToString()); 

        Datos d = new Datos(lap, lista);                                                                         

        //Cçreamos el motor de entrenamiento, donde se le indican los datos a entrenar el training rate y el tamaño del batch
        BrightWire.IGraphTrainingEngine engine = graph.CreateTrainingEngine(d, TRAINING_RATE, 128);            
        //esto sirve para ajustar el training rate cada cierto número de batchs, no lo uso ahora mismo.
        engine.LearningContext.ScheduleLearningRate(100, TRAINING_RATE / 2);                        

        //métrica de error o función de pérdida, cuadrática (mse supongo en el estándar de keras)
        var errorMetric = graph.ErrorMetric.Quadratic; 

        // create the network
        graph.Connect(engine)
            .AddFeedForward(outputSize: 4)  //tipo de capa
            .Add(graph.ReluActivation()) // tipo de activación.  
            .AddFeedForward(outputSize: 1)   //última capa, indicamos el tamaño de salida el del trainingdata
            .Add(graph.ReluActivation())
            .AddBackpropagation(errorMetric)
        ;            

        // train the network for twenty iterations, saving the model on each improvement                        
        for (int i = 0; i < 1000; i++)
        { 
            System.Diagnostics.Debug.WriteLine (engine.Train(context).ToString()); //la salida de esto parece ser el error total (loss)                                            
        }

        var resultado = engine.Execute(new float[] { 1f, 0f, 1f });  //probamos a ejecutar a ver cómo nos ha quedado.
        MessageBox.Show(resultado.Output[0].ToString());
        resultado = engine.Execute(new float[] { 1f, 1f, 1f });
        MessageBox.Show(resultado.Output[0].ToString());

        return null; 

    }

I cannot see the code that is raising the error..... Any hints about what is going on?

Thanks for your patience.

[jdermody]

I can see the problem, the data source that you've copied is actually designed for executing results and not training data (sorry!). Take a look at VectorBasedDataTableAdaptor for a better example. (The problem is the null parameter here that should contain the classification targets for the current mini batch:

return new MiniBatch(rows, this, inputList, null);

[Abner77]

Hi jdermody, I have changed the code to put the outputs, if that's what you meant, however, is not working (the network does not return correct values). As far as I go, there's something I'm not understanding here which I think, is related to the problem, may be?

var data = rows.Select(i => _data[i]).ToList(); var input = _lap.CreateMatrix(data.Count, InputSize, (x, y) => data[x].Data[y]); var inputList = new List {new MatrixGraphData(input)}; var output = _lap.CreateMatrix(data.Count, OutputSize, (x, y) => data[x].Data[y]); var outputList = new MatrixGraphData(output) ;

        return new MiniBatch(rows, this, inputList, outputList);

The MiniBatch constructor gets as inputList a List and as output a GraphData, so, it seems counterintuitive for me., as inputList has just one element and inside a MatrixGraphData, while outputLIst gets 8 elements (rows) because it's a MatrixGraphData, not inside a list. Is this correct? The code runs now, but, as I say, no matter what i do, the result is not the output it should be.....

I've implemented a simple double XOR operation. sb.AppendLine("0,0,0,0"); sb.AppendLine("0,0,1,1"); sb.AppendLine("0,1,0,1"); sb.AppendLine("0,1,1,0"); sb.AppendLine("1,0,0,1"); sb.AppendLine("1,0,1,0"); sb.AppendLine("1,1,0,0"); sb.AppendLine("1,1,1,1");

var resultado = engine.Execute(new float[] { 1f, 0f, 1f }); //testing MessageBox.Show(resultado.Output[0].ToString()); resultado = engine.Execute(new float[] { 1f, 1f, 1f }); //testing MessageBox.Show(resultado.Output[0].ToString());

//both operations show always the same result (either 0 or 1) when the first one should be 0 and the second one 1.

The model at this point is

// create the network graph.Connect(engine) .AddFeedForward(outputSize: 16) //tipo de capa .Add(graph.ReluActivation()) // tipo de activación.
.AddFeedForward(outputSize: 1) //última capa, indicamos el tamaño de salida el del trainingdata .Add(graph.ReluActivation()) .AddBackpropagation(errorMetric) ;

        for (int i = 0; i < 10000; i++)
        { 
            System.Diagnostics.Debug.WriteLine (engine.Train(context).ToString()); //la salida de esto parece ser el error total (loss)                                            
        }

One thing I've noticed debugging the code is that, each time this method is called in my DataSource ..... public void OnBatchProcessed(IContext context) { // nop }

the context.Trainingerror shows always around 0.7, no matter how many times this method is called.

Any hints? Sorry for bothering you and thanks for your time

[jdermody]

No problem, happy to help!

It looks like you might be creating a bad output vector by mistake - your code seems to be copying the same value into both the input vector and output vector. Try:

var data = rows.Select(i => _data[i]).ToList();
var input = _lap.CreateMatrix(data.Count, InputSize, (x, y) => data[x].Data[y]);
var output = _lap.CreateMatrix(data.Count, OutputSize, (x, y) => data[x].Data[y + InputSize]);
var inputList = new List<IGraphData> {new MatrixGraphData(input)};
return new MiniBatch(rows, this, inputList, new MatrixGraphData(output));

I ran your code with that change and got the following, which looks like a better result:

[Abner77]

Thanks!! It worked!

[jdermody]

It should be easier to reuse BW components as most are now public. Changes merged with master and will be available in the forthcoming 2.1 release