ConvolveFilterBankToFilterBank

[jfrancis71]

Note if our convolutions are occurring in spatial domain we are likely to be fairly efficient at present.

Opportunity for speed up in ConvolveFilterBankTo2D as linear sum. This would save a few function calls.

If we are computing in the frequency domain we are recomputing fourier transforms of data unnecessarily. I suspect this is not the case.

If we are in spatial domain, it is possible that some "housekeeping" tasks are being repeated unnecessarily for each new feature map, e.g. if implemented by partitioning data then this is being done repeatedly. Note the dot prod calculations would remain same.

[jfrancis71]

Have explored ConvolveFilterBankTo2D as a single ListCorrelate call. It can be done, but it is slightly slower, around 0.1 secs versus .078 with the explicit per kernel correlation and sum. So not implemented.

Note our spatial equivelant is around 100 * 12 * 12 * 5 * 5 * 32 / (.078 secs) = 147M inner element prods Assuming each needs a multiplication and addition takes you to 300MFLOPS Add in some housekeeping and that is a reasonable performance for an old laptop. Therefore closing this off.

[jfrancis71]

Note reimplementing using dot prod method. Attribute to paper Looking good, do a transpose so that 32 is on top then table To add biases and then re transpose back to normal format

[jfrancis71]

Possible interesting paper on frequency domain convolution: http://bmvc2015.swansea.ac.uk/proceedings/papers/paper160/paper160.pdf Very Efficient Training of Convolutional Neural Networks using Fast Fourier Transform and Overlap-and-Add U.S. Air Force Research Laboratory

[jfrancis71]

https://etd.ohiolink.edu/!etd.send_file?accession=wright1432747175&disposition=inline Neural Networks using Fast Fourier Transform Tyler Clayton Highlander (2014)

[jfrancis71]

We can do the entire convolution for all filter banks in one call: ListCorrelate[ Table[{kerns[[l]]}, {l, 1, 5}], {dat}, {-1, 1, 1, 1, 1}, {-1, -1, -1, -1, -1}];

This doesn't work well at all. I think it ends up computing a sliding window of the top level kernels, so they end up being recomputed many times.

[jfrancis71]

Fourier domain:

z = Table[Total[Fkerns[[l]]*o], {l, 1, 32}]; // AbsoluteTiming is around .87 secs. too slow

where Fkerns is the 32_32 array of 20_20 in frequency domain and o is 32_20_20*100 in frequency domain

This is a bit disappointing, only 26MOps. Well, 50MFLOPS. Note, tricky to make faster, not a matrix multiplication as the dot prod changes on both left and right side

[jfrancis71]

Paper by Le Cun et al: (2013) http://arxiv.org/pdf/1312.5851v5.pdf

Also: http://mathematica.stackexchange.com/questions/17758/ways-to-compute-inner-products-of-tensors