Open CarloLucibello opened 3 years ago
related issues I could find are #1307 #1273
Maybe this should use Dense(n, n, relu)
, as gradient(sum, rand(Float32, 3))[1] isa Zygote.Fill
which I think gets you a generic *
(but wouldn't happen in real use), while gradient(x->sum(relu, x), rand(Float32, 3))[1] isa Array
. This shaves off an order of magnitude at n=200
.
Good catch! I'm updating the script with relu
(and the output as well).
As you said, that FillArrays performance problem is not relevant in our real scenarios. Nonetheless, I wrote a fix here https://github.com/JuliaArrays/FillArrays.jl/pull/129
This is just to track the performance of the Dense layer on CPU. I use the following script:
and on my system:
I obtain the following output
Some observations:
grandient
's calling styles.