Autodiff support for sorting + numerical der checking

[jotaf98]

Awesome stuff, thanks!

[ryanwebster90]

yay sorting! been on my to do list for a while.

[ryanwebster90]

Do you think it might be worthwhile to wrap sort with something like this?

gpu  =false;
if isa(x,'gpuArray')
x = gather(x);
gpu = true
end
y = sort(x,dim);
if gpu
x = gpuArray(x);
end

It's about 2x faster for many inputs on my machine.

[jotaf98]

That's true, although it depends on the input size! I think this should be controlled explicitly. You can already use gather and gpuArray to define at which points you want to send something to/from the GPU. So in this case:

x = Input();
...
x_cpu = gather(x);
y_cpu = sort(x_cpu);
y = gpuArray(y_cpu);

Note that sometimes you may want to keep things on the CPU from that point onwards, or not do it at all. For example, I found that highly serial operations like customized RNNs on small vectors are faster on the CPU, even if the embeddings themselves originate on the GPU (e.g. from a CNN).