How to implement a time-delay neural network using RSNNS?

[heathyates]

I do not see where time-delay neural network is documented? Can you please be so kind and gracious enough to let me know where the tdnn is documented in your package documentation. If not, could you be so kind and gracious enough to briefly include an example of how to do this with the SNSS code please?

I have read an article about SNNS package from 2012. The authors original article is here. They mention on page 2 that several network architectures are implemented in SNSS, including time-delay neural networks. I believe the source code is located here. Unfortunately, when I look at the R package documentation I do not see this mentioned in the package api.

[cbergmeir]

Hi, yes, good question. I never used tdnn networks in RSNNS myself and also don't know anybody who has. I.e., it should be supported but was never tested and used. The most useful information you'll find is in inst/KnownIssues. There, it says:

As RSNNS has a 3-level architecture:

RSNNS high-level api (rsnns) RSNNS low-level api (SnnsR) C++ api (SnnsCLib)

methods might be implemented on an architecture level, but not on the next one above. E.g. for hopfield and td networks, there is currently no convenient create function for the networks in RSNNS. However, these can be created with the original bignet tool and then loaded into RSNNS, or they could be created using low-level functionality to create units and links. Such a "manual" creation using SnnsR functions is shown e.g. in the encoder demo. If you are interested in a particular type of network, don't hesitate to make a feature request.

This means, all the necessary functions are included in RSNNS, and it should be possible to write a working example with the RSNNS low-level interface. However, I've never written such an example. Also, as I didn't include the SNNS code for generating such a network, you'd either need to include the code yourself in RSNNS (I could help with that) or you'd need to implement it yourself with the low-level interface. As said, the "encoder" demo is probably the closest you get.

[heathyates]

@cbergmeir Thank you for your reply. Please let me repeat the above to you to make sure I understand. Am I understanding the above correctly? If not, please be so kind and gracious enough to correct me. Your reply can be summarized in three parts:

• There is no high level TDNN api for TDNN
• It should be possible to implement TDNN using SnnsR, but there is no documentation
• It should be possible to implement TDNN using RSNNS and you are willing to assist in that

You mentioned you could help me with trying to include code myself in RSNNS. I'm good with that if you can help. I'm a software developer in C# and javascript primarily, but I have some experience with R and C++ as I am wanting to evolve into machine learning further. As a software developer, this is how I approach things and would like to approach this problem (please let me know if there is a better way):

• Implement a super easy, hello world type TDNN network in SNNS
• Generalize the above for generating a network in SNNS
• Include that code in SNNS so we can eventually use RSNNS high level api?

If so, can you help me with this and guide me? I'm willing to do a ton of work on this, but to get started I need help. First, I need to know what are good documentation so I can even do the hello world. I need to familiarize myself with the tool and all that. Second, I'll need guidance on the generalization part and flat out collaboration on the third part so other folks can benefit from all of this in the future. I've seen a lot of folks asking around about TDNN on forums and there really isn't anything out there, so if you're willing to collaborate with me on this, I think it might be a lot of fun and benefit the community? I'm willing put in a lot of time and effort on this, but need someone to keep me from falling off a cliff or getting stuck. Thoughts sir?

Heath

[cbergmeir]

Hi,

as I said before, have a look at the encoder demo to see how to use the very low level API. The two things about TDNN that you need are 1. generate the network 2. train it.

The second part should be pretty straightforward. Look at the SNNS user manual section 9.10.2 from there you will see that what you will get the activation function, update function, etc. to use (just replace them in the encoder example). The parameters of TimeDelayBackprop are explained on page 68 of that document.

The first part will be a bit more work. Have a look at the /src folder. You'll see the files that start with bn_. You need to implement one of those for TDNN. Get the original SNNS sources from here. I then also suggest you compile them, run SNNS to see the original GUI and have a look at the original BIGNET tool. Have a look at the subfolder /SNNSv4.3/xgui/sources/ . There, you'll find a file bn_TD_bignet.c that you need to adapt. Best thing to follow is probably to look at, e.g., bn_JordElm, to see how it was adapted and how it is used in RSNNS (all the way up to the high-level API). You want to do the same with bn_TD_bignet. Hope this helps to get you started.

[heathyates]

Hi,

Thanks for your reply. Your reply definitely gets me started I think and I appreciate the patient effort to include links. That was very useful. As you said above, the first part will be some more work. This is going to be quite a bit of work, but I'll be in touch as I go here. Shall we leave this open indefinitely and update as I progress along?

Thanks, Heath

[cbergmeir]

Yeah, sure, just let me know how you go and if you get stuck. Also, maybe just fork RSNNS here on github and put your changes in there