Model to predict result of sine function

[rychuelektryk]

Hi,

I'm trying to learn KoltinDL api and I thought that as a first step I could use it to create simple NN model that predicts result of a sine function.

Is it even possible in current version of KotlinDL? It seems like the simpliest model one could try to create, bu I have a feeling that KotlinDL only supports classification models, and no regression models. Am I right?

It seems that predict functions returns only probability of a predicted label instead of a predicted sine value.

My code:

`class DlModelCreator : IDlModelCreator { override suspend fun createModel() { val input = prepareInput()

    val (train, test) = input.split(0.8)

    val model = Sequential.of(
        Input(1),
        Dense(10),
        Dense(1, Activations.Linear)
    )

    model.use {
        it.compile(
            optimizer = Adam(),
            loss = Losses.MAE,
            metric = Metrics.MAE
        )

        it.summary()

        it.fit(
            dataset = train,
            epochs = 10,
            batchSize = 100
        )

        val evaluationResult = it.evaluate(dataset = test, batchSize = 100)//.metrics[Metrics.MAE]

        log("evaluationResult: $evaluationResult")

        repeat(test.xSize()) { id ->
            val xReal = test.getX(id)
            val yReal = test.getY(id)

            val yPred2 = it.predict(xReal) // always returns 0
            val yPred3 = it.predictSoftly(xReal) // returns value oscillating around 1.0

            log("xReal: ${xReal[0]}, yReal: $yReal, yPred2: $yPred2, yPred3: $yPred3")
        }
    }
}

private fun prepareInput(): Dataset {
    val sampleCount = 100000

    val x = Array(sampleCount) { FloatArray(1) }
    val y = FloatArray(sampleCount)

    repeat(sampleCount) {
        val xSample = Random.nextDouble(0.0, Math.PI * 2).toFloat()
        val ySample = sin(xSample)

        x[it][0] = xSample
        y[it] = ySample
    }

    val result = OnHeapDataset.create(x, y)

    return result
}

}`

[zaleslaw]

Hi @rychuelektryk You asked a good question, the Linear Regression works, for example (an example)

So, I see the problem with the predict/predictSoftly usage, give me a few days to understand what's happened and what should be changed in the 0.3 release for regression tasks.

Did you copy this example from Keras? Could you please share the Python code?

[rychuelektryk]

Hi @zaleslaw and thanks for the answer. The keras equivalent would be something like this:

` import numpy as np from keras.models import Sequential from keras.layers import Dense from tensorflow.keras.optimizers import Adam import random import math

def prepareInput():
    sampleCount = 100000

    x = np.random.rand(sampleCount, 1)

    x *= math.pi * 2

    y = np.sin(x)

    xTrain, xTest = np.split(x, 2)
    yTrain, yTest = np.split(y, 2)

    return xTrain, yTrain, xTest, yTest

xTrain, yTrain, xTest, yTest = prepareInput()

model = Sequential()
model.add(Dense(10, input_shape=(1,),activation = 'relu'))
model.add(Dense(1,activation = 'linear'))
model.compile(loss='mean_squared_error', optimizer=Adam(), metrics=['mean_squared_error'])

history = model.fit(xTrain,yTrain, epochs=10, batch_size=100)

prediction = model.predict(xTest)

for i in range(10):
    print(f"input: {xTest[i]}, predicted: {prediction[i]}, target: {yTest[i]}")

`

Epoch 1/10 500/500 [==============================] - 1s 821us/step - loss: 0.3736 - mean_squared_error: 0.3736 Epoch 2/10 500/500 [==============================] - 0s 833us/step - loss: 0.1797 - mean_squared_error: 0.1797 Epoch 3/10 500/500 [==============================] - 0s 834us/step - loss: 0.1548 - mean_squared_error: 0.1548 Epoch 4/10 500/500 [==============================] - 0s 847us/step - loss: 0.1520 - mean_squared_error: 0.1520 Epoch 5/10 500/500 [==============================] - 0s 843us/step - loss: 0.1506 - mean_squared_error: 0.1506 Epoch 6/10 500/500 [==============================] - 0s 835us/step - loss: 0.1494 - mean_squared_error: 0.1494 Epoch 7/10 500/500 [==============================] - 0s 835us/step - loss: 0.1481 - mean_squared_error: 0.1481 Epoch 8/10 500/500 [==============================] - 0s 836us/step - loss: 0.1469 - mean_squared_error: 0.1469 Epoch 9/10 500/500 [==============================] - 0s 835us/step - loss: 0.1457 - mean_squared_error: 0.1457 Epoch 10/10 500/500 [==============================] - 0s 835us/step - loss: 0.1447 - mean_squared_error: 0.1447 input: [4.92054105], predicted: [-0.6050269], target: [-0.97841446] input: [2.93097985], predicted: [0.1802252], target: [0.2090592] input: [3.40129467], predicted: [-0.00540149], target: [-0.25679257] input: [0.975655], predicted: [0.679202], target: [0.82806927] input: [1.63460678], predicted: [0.69188535], target: [0.9979648] input: [0.19171183], predicted: [0.45147228], target: [0.19053964] input: [1.93774242], predicted: [0.57224184], target: [0.93342734] input: [5.45218234], predicted: [-0.8148579], target: [-0.73860788] input: [5.26077695], predicted: [-0.7393133], target: [-0.853366] input: [1.00824174], predicted: [0.68866825], target: [0.84589538]

The result is poor, but the result of predict function is what I expected to see.

In case of code written with KotlinDl the model seems to be learning well because evaluation shows mae near 0. It seems that the problem is only with predict function which always returns 0, and predictSoftly function which returns ~1.0

[zaleslaw]

Yes, I've run and debugged your example. Thanks for this; I hope to fix it and share the results here in the comments

[zaleslaw]

@rychuelektryk thanks for trying KotlinDL, I fixed a label encoding, added your example with sin function and it will available in release 0.3 and its preview since 1 september

Also regression works well on the 0.1.1 version from maven central (dataset constructor is little bit different) because it was broken in 0.2 only

[rychuelektryk]

@zaleslaw Great! I really appreciate your work. Keep it up and thanks ;-)