search

ixaxaar / pytorch-dnc

Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch
MIT License
335 stars 56 forks source link

Feature: Sparse Access Memory #4

Closed ixaxaar closed 6 years ago

ixaxaar commented 6 years ago

Implement sparse access memory for DNCs. Related paper: https://arxiv.org/abs/1610.09027

MaxwellRebo commented 6 years ago

Are you already diving into this, or do you need some help?

ixaxaar commented 6 years ago

Not looking at this at the moment. Trying to make DNCs work for language modeling & seq2seq.

MaxwellRebo commented 6 years ago

I've done DNCs for seq2seq before. Does anything about the base implementation need to change, or is it just a matter of structuring the inputs and outputs?

ixaxaar commented 6 years ago

I actually wrote this repo because the other ones were not torch.nn-like. Making the APIs more like torch.nn makes it easier to replace nn.LSTM etc with DNC otherwise keeping implementations intact, potentially requiring one line of code change.

The ones I tried, I had to also write this piece of code for LMs to detach hidden states: https://gist.github.com/ixaxaar/02b65312eb1a840cc14e74d93b35178f For seq2seq I took the openNMT repo and forked it here.

Other than that it was pretty much a few liners.

ixaxaar commented 6 years ago

Is implemented in #16 and #18 , though code review etc are pending.