search

nvecoven / BRC

A repository containing the code for the Bistable Recurrent Cell
47 stars 6 forks source link

Pytorch Implementation of BRC with Additional Test on Binary Addition #4

Open niklexical opened 4 years ago

niklexical commented 4 years ago

Hi, just wanted to say impressive work on the cells. Since it currently benefits the Tensorflow community, Jannis and I developed a Pytorch version of the BRC and nBRC, and tested it on benchmark 1. It would be great if you could have a look at the code and let me know if you have any suggestions/spot any bugs, so the Pytorch community could also make use of your research. Feel free to also point Pytorch users looking to implement your ideas to that repository.

Something that really interested us was how the cells performed on Binary Addition. Since it has been established that standard recurrent cells face difficulties in discrete counting tasks, particularly for longer sequences (see 1, 2, 3 ), we additionally evaluated the performance of these cells on the Binary Addition task. The results illustrate the consistently better performances of the BRC and nBRC over the GRU and LSTM cells, thus highlighting another scenario where these cells excel.

We're curious as to how you would hypothesise this behaviour. Do you think it is primarily due its capacity for long lasting memory, and you expect it to do well in these kind of tasks? or is it a pleasant surprise to you as it was to us?

nvecoven commented 4 years ago

Hi,

Thanks for the feedback, and great work on the Pytorch implementation, I will immediately update the readme and include a link to your repository so others can benefit from it !

It is a pleasant surprise to us as well. It is hard to pinpoint exactly what is happening. It might indeed be thanks to the long lasting memory, however one of our guess is that the bistability precisely allows to remember binary values (high and low stable states) which seems like it would be well-suited for such a task. In both cases, it is indeed an interesting find !

jannisborn commented 4 years ago

Many thanks for the fast cross-referencing!

Interesting. Would your intuition be that the bistability facilitates some form of "exact memory" as opposed to the rather fuzzy memory of vanilla RNN cells?