Open NHDaly opened 5 years ago
The Result type represents the result of an operation that is recorded for gradient calculation. If a function has zero gradient (i.e. no dependence on the input), it does not need to get recorded, therefore it does not create a Result. similar
is an example, so are size
, length
, eltype
etc.
Ideally this should all be transparent to the programmer. The only functions in the API open to the programmer are Param, @diff, value and grad (i.e. I should be able to change the name or fields of Result without breaking any of your code).
Type stability is a difficult topic: while running outside of a @diff context, all Params are ignored and no Results are generated eliminating most of the overhead of AutoGrad. This means the same variable could be Array during some calls and Result{Array} during others.
In your case if you want to prevent the type change inside the for loop it may be possible to do term = 0 * coef[:,ndim]
in the beginning.
Hi! :) I think I've stumbled across a correctness issue w/ the
similar
function that leads to performance problems:As far as I can tell, even when actually running within an
@diff
,similar(AutoGrad.Result{Array})
returns anArray
. I would think, instead that it should return aResult{Array}
.I came across this trying to help a coworker diagnose why their AD code is slow. I saw from
@btime
that their code has a lot of allocations, which seem to grow with the size of the data (so it's not a one-time overhead related to compiling the extra methods).I used
Cthulhu.jl
to look through the@code_warntype
of the functions being differentiated, and found this:That seems surprising to me! They were trying to use
similar
to prevent type-instability in this for-loop:As the comment explains, first we had
term = zeros(ndim)
, but were concerned about the type instability inside the for-loop (the type ofterm
would change after adding theResult{Array}
to it). So we tried usingsimilar(..., 0)
to create aResult{Array}
to start with so that addingcoef
to it doesn't change its type. But surprisingly, this returned the same thing aszeros()
.Shouldn't
similar(::AutoGrad.Result{Array{Float64,1}})
also return anAutoGrad.Result{Array{Float64,1}}
, or am I misunderstanding? Thanks! :slightly_smiling_face: