[FEATURE] General state detection

[palisn]

Add a new model that predicts the current state of the imaginary keyboard. Optimally, this may introduce combined press and release detection and additional row detection in one go.

This depends on the data collection including multiple fingers and rows to work somewhat reliably. Thus, it depends on the issue #3.

[palisn]

Issue #3 has now been resolved. After some data is collected, this can be actively worked on.

[palisn]

Following sub-issues that are relevant to this issue were recently introduced:

• 28

• 29

[palisn]

For testing purposes, we established a moderately large neural network consisting of four hidden layers, each 5000 neurons with the ReLU activation function, and a 17-neurons large output layer with sigmoid activation function. Additionally, each layer has L2 regularization.

model = keras.Sequential(
    [
        layers.Input(shape=(179,)),
        layers.Dense(5000, activation = "relu",
                     kernel_regularizer=regularizers.l2(0.001)),
        layers.Dense(5000, activation = "relu",
                     kernel_regularizer=regularizers.l2(0.001)),
        layers.Dense(5000, activation = "relu",
                     kernel_regularizer=regularizers.l2(0.001)),
        layers.Dense(5000, activation = "relu",
                     kernel_regularizer=regularizers.l2(0.001)),
        layers.Dense(17, activation = "sigmoid")
    ]
)

We compiled the model with a BinaryCrossentropy loss – which is typically used for multi-label problems – and the RMSProp optimizer. Afterward, we trained the model with a batch size of 32 for 100 epochs (excluding early stopping).

The results show that the model does converge but yields terrible results for our positive class; thus, it does not perform well regarding our metric.

Learning Rate	Loss	Metric	Convergence
1-e3	0.03	0.29
5-e4	0.03	0.29
1-e4	0.03	0.43
5-e5	0.025	0.44

(Warning: The x axes do not necessarily start at 0 to increase readability.)

This emphasizes the issue that the dataset is unbalanced. For better results, we should probably change the model architecture or introduce some counter-measure against the issue.

29

[palisn]

To address the largely different domains of the features, we introduced Normalization (also known as Standardization) into the model:

inputs = layers.Input(shape=(179,))
normalizer = layers.Normalization()
normalizer.adapt(x_train)
x = normalizer(inputs)
for _ in range(4):
    x = layers.Dense(5000, activation = "relu", kernel_regularizer=regularizers.l2(0.001))(x)
outputs = layers.Dense(17, activation = "sigmoid")(x)
model = keras.Model(inputs, outputs)

This change caused visible improvement. With a learning rate of 1-e4 and 100 epochs of training (the same setup as in the last message) we were able to achieve:

• a lower loss: 0.025
• a better metric score: 0.55
• better results for the positive class; $f_2$-score: 0.3 (as opposed to 0.2 before)
• still a bit more potential for further improvement with further training

[palisn]

still a bit more potential for further improvement with further training

After 315 epochs there is no more improvement and the result is:

• loss: 0.021
• metric score: 0.69
• $f_2$-score: 0.5

Quite the improvement and validation shows that we are not overfitting. But, it is still not enough to be usable in practice.

[palisn]

As an alternative to BinaryCrossentropy we tried BinaryFocalCrossentropy as it is designed to help learn unbalanced datasets. We were able to achieve a loss slightly below 0.01. Surprisingly, this did not reflect on our metric or on results for positive class in general.