Closed mazko closed 5 years ago
If I am right, you can replace kad_avg(1, &t)
with kad_select(1, &t, -1)
, which selects the last hidden vector. I haven't touched kann for a while, so I could be wrong...
kad_node_t *t;
int rnn_flag = KANN_RNN_VAR_H0;
if (norm) rnn_flag |= KANN_RNN_NORM;
t = kann_layer_input(28); // 28x28
for (i = 0; i < n_h_layers; ++i) {
t = kann_layer_gru(t, n_h_neurons, rnn_flag);
t = kann_layer_dropout(t, dropout);
}
// replace "t = kad_avg(1, &t);" with:
t = kad_select(1, &t, -1);
ann = kann_new(kann_layer_cost(t, 10, KANN_C_CEB), 0);
i am not sure how to train such network.
When you use kad_avg()
or kad_select()
, there will be a single output of shape (batch,10)
in case of MNIST (independent of unrolled length ulen
). The following part needs to be changed to:
n_var = kann_size_var(ann);
r = (float*)calloc(n_var, sizeof(float));
x = (float**)malloc(d->ulen * sizeof(float*));
y = (float**)malloc(1 * sizeof(float*));
for (j = 0; j < d->ulen; ++j)
x[j] = (float*)calloc(mini_size * d->n_in, sizeof(float));
y[0] = (float*)calloc(mini_size * d->n_out, sizeof(float));
shuf = (int*)calloc(d->n, sizeof(int));
each y in sequence of output has same value d->y[s]
KANN only uses the first d->y[0]
, so your training is still successful.
It would be great to see any simple rnn classification example.
Yes, indeed. I should add such an example at some point...
Thank you for clarifying. I changed a little bit train
function according to your suggestions.
static void train(kann_t *ann, train_data *d, float lr, int mini_size, int max_epoch, const char *fn, int n_threads)
{
float **x, **y, *r, best_cost = 1e30f;
int epoch, j, n_var, *shuf;
kann_t *ua;
n_var = kann_size_var(ann);
r = (float*)calloc(n_var, sizeof(float));
x = (float**)malloc(d->ulen * sizeof(float*));
y = (float**)malloc(1 * sizeof(float*));
for (j = 0; j < d->ulen; ++j) {
x[j] = (float*)calloc(mini_size * d->n_in, sizeof(float));
}
y[0] = (float*)calloc(mini_size * d->n_out, sizeof(float));
shuf = (int*)calloc(d->n, sizeof(int));
ua = kann_unroll(ann, d->ulen);
kann_set_batch_size(ua, mini_size);
kann_mt(ua, n_threads, mini_size);
kann_feed_bind(ua, KANN_F_IN, 0, x);
kann_feed_bind(ua, KANN_F_TRUTH, 0, y);
kann_switch(ua, 1);
for (epoch = 0; epoch < max_epoch; ++epoch) {
kann_shuffle(d->n, shuf);
double cost = 0.0;
int tot = 0, tot_base = 0, n_cerr = 0;
for (j = 0; j < d->n - mini_size; j += mini_size) {
int b, k;
for (b = 0; b < mini_size; ++b) {
int s = shuf[j + b];
for (k = 0; k < d->ulen; ++k) {
memcpy(&x[k][b * d->n_in], &d->x[s][k * d->n_in], d->n_in * sizeof(float));
}
memcpy(&y[0][b * d->n_out], d->y[s], d->n_out * sizeof(float));
}
cost += kann_cost(ua, 0, 1) * d->ulen * mini_size;
n_cerr += kann_class_error(ua, &k);
tot_base += k;
//kad_check_grad(ua->n, ua->v, ua->n-1);
kann_RMSprop(n_var, lr, 0, 0.9f, ua->g, ua->x, r);
tot += d->ulen * mini_size;
}
if (cost < best_cost) {
best_cost = cost;
if (fn) kann_save(fn, ann);
}
fprintf(stderr, "epoch: %d; cost: %g (class error: %.2f%%)\n", epoch+1, cost / tot, 100.0f * n_cerr / tot_base);
}
kann_delete_unrolled(ua);
for (j = 0; j < d->ulen; ++j) {
free(x[j]);
}
free(y[0]); free(y); free(x); free(r); free(shuf);
}
Both previous and current implementations produce exactly the same mnist-rnn.kan file.
I also tried kad_select
instead of kad_avg
and it seems first one learns faster on mnist.
~$ ./mnist-rnn -o mnist-rnn.kan kann-data/mnist-train-?.knd
~$ ./mnist-rnn -i mnist-rnn.kan kann-data/mnist-test-x.knd | kann-data/mnist-eval.pl
Error rate: 2.45% # kad_avg(1, &t);
Error rate: 1.42% # kad_select(1, &t, -1);
Thanks a lot for the confirmation. Do you mind if I modify your code a little bit and add to the "examples/" directory? I will say something like "Code was written by Oleg Mazko (@mazko) and modified by Attractive Chaos". Is that ok?
Thanks a lot for the confirmation. Do you mind if I modify your code a little bit and add to the "examples/" directory? I will say something like "Code was written by Oleg Mazko (@mazko) and modified by Attractive Chaos". Is that ok?
Of course it's ok.
There is also a project https://github.com/42io/c_keyword_spotting which is based on kann. I am planing to add rnn there too in a couple weeks or so.
Thanks a lot for the confirmation. Do you mind if I modify your code a little bit and add to the "examples/" directory? I will say something like "Code was written by Oleg Mazko (@mazko) and modified by Attractive Chaos". Is that ok?
Of course it's ok.
There is also a project https://github.com/42io/c_keyword_spotting which is based on kann. I am planing to add rnn there too in a couple weeks or so.
Have you finished this goal to applay GRU in Kann? @mazko
When classify a sequence, we would like the network to have one output, instead of a sequence of output. According to 01user.md to classify a sequence
kad_avg
was mentioned. I tried this on mnist. It works but i am not sure how to train such network. During training process we don't even know output values other then last one. In this linememcpy(&y[k][b * d->n_out], d->y[s], d->n_out * sizeof(float));
eachy
in sequence of output has same valued->y[s]
which looks strange.It would be great to see any simple rnn classification example.