dfdx
commented
6 years ago Let me explain these issues and how to fix them.
ERROR: MethodError: no method matching parse!(::Espresso.ExGraph, ::Espresso.ExH{:+=})
As you correctly assumed, XGrad (or, more specifically, Espresso) can't parse +=. But there's a serious reason for this: Espresso assumes expressions to be mathematically correct, and in math there can't be any mutation. So rewriting it as:
ℓ = 0.0
ℓ = ℓ + logpdf(Normal(0, 5), μ)doesn't help since you still modify ℓ.
The easiest way to fix it is to use new names for variables, e.g. something like:
μ = θ[1]
ℓ = logpdf(Normal(0, 5), μ)
σ = softplus(θ[2])
ℓ2 = ℓ + abs(σ - θ[2])
ℓ3 = ℓ2 + logpdf(Truncated(Cauchy(0, 3), 0, Inf), σ)
...I thought about automatic renaming so that you don't have to do it manually, but implications are unclear at the moment.
For the same reason there's still no first-class support for loops in Espresso - without mutating variables loops are nearly useless (I think we will add support for them anyway, but design depends on use cases, and I haven't found a good one yet). Your example may be rewritten (automatically, so that the user doesn't have to worry about it) into something like:
d = ...
ℓ_full = sum(logpdf.(d, x))I'm also worried about distribution types like Normal or Cauchy - although it's convenient to use a custom type to represent distribution in Julia, it's not something automatic differentiation can work with. For example:
N = Normal(0, 5)
ℓ = logpdf(N, μ)If ℓ is a number, dℓ/dμ is also a number. But what is dℓ/dN? In some cases, XGrad can dive into a structure of Normal type and correctly derive derivatives of its parameters, but we might need to use a more functional approach (e.g. normlogpdf). Anyway, let's see how it works.
I propose to go in 2 steps:
- Create an expression that is actually differentiable, i.e. can be parsed and includes only defined functions with known derivatives.
- Write transformation that takes user input in a convenient form (e.g. with loops, variable mutations or whatever) and produces an expression for differentiation.
Does it sound reasonable to you?
cscherrer
I'm trying to use
XGradfor Soss, and running into some problems.Starting with something like this:
I get
I'm not sure how to teach it about
+=, so I change to this:That helps a little:
I thought maybe I could use
normlogpdfinstead, but that's not any better: