Definition and implementation of 'Loss' in Linear Regression Tutorial "Julia & Lux for the Uninitiated"

CMoebus commented 1 month ago

Thanks for the tutorial, which has in some places a steep learning curve (at least for me ;) ). I had some problems with the definition of the loss function 'mse' and the implementation of it later.

In the LaTex-formula for the error-sums-of-squares the division is by 1/2. This is an error if you define the mse. It should be 1/N. Furthermore later on the function mse is implemented by two methods:

function mse(model, ps, st, X, y) y_pred, st_new = model(X, ps, st) return sum(abs2, y_pred .- y), st_new end
mse(weight, bias, X, y) = sum(abs2, weight * X .+ bias .- y).

But despite their function name 'mse' they only implement the usual 'ssq' (=sum-of-squares). I think this should be corrected. Besides that is the implementation of the iteration loop a bit opaque. At least for one new to Lux.jl and DL.

Thanks a lot in advance, Claus

avik-pal commented 1 month ago

right, they should be mean(abs2, ...). Make a PR?

Besides that is the implementation of the iteration loop a bit opaque

Do you have specific suggestions for improvement?

CMoebus commented 1 month ago

Hi Avik, I 've made some refactorings within Pluto-notebooks. Because Pluto has a declarative style I introduced some functions. Here are two links. One, to a new t-test with two submodels and the second to the refactored 'linear regression' tutorial. Maybe there are more elegant solutions, but then I need more understanding of the workings of LUX.jl.

All the best, Claus

LuxDL / Lux.jl

Definition and implementation of 'Loss' in Linear Regression Tutorial "Julia & Lux for the Uninitiated" #664