search

JuliaDiff / ChainRulesCore.jl

AD-backend agnostic system defining custom forward and reverse mode rules. This is the light weight core to allow you to define rules for your functions in your packages, without depending on any particular AD system.
Other
257 stars 62 forks source link

Destination flipping for `add!!(x, y)` #346

Open oxinabox opened 3 years ago

oxinabox commented 3 years ago

Right now add!!(x, y) will only ever mutate x, never y. because of the logically linear nature of AD, y being a differential (i.e. the output of pullback) means it is used exactly once . Which means we are infact free to mutate it, also since this will be it's only use. (same also says why we are always ok to mutate x) See https://github.com/google-research/dex-lang/issues/401#issuecomment-756996758 (and to a lesser extent https://twitter.com/oxinabox_frames/status/1387722380978622468)

So we could pick either x or y depending on which was a valid inplacable_destination. And maybe also always chose Array if one of them is an Array (since that is the best destination if that has been allocated)

This would solve at least the basic cases of #345

oxinabox commented 3 years ago

Argument agaist this is that if y comes from identity_pullback(y)=y and was provided by the user then the user might be using it for something else also.

Kind feel like that shouldn't be our problem though.